JP2020108099A - Electronic apparatus and method for controlling the same - Google Patents

Abstract

To create, from a VR content, a VR content in a narrower range, and make it possible to more easily check a video range of the VR content after cut-out.SOLUTION: An electronic apparatus of the present invention controls display means to display, on a first screen, a partial range of a VR content having a first video range as a display range, changes the position of the display range according to a change in attitude of the electronic apparatus or a display range change operation performed by a user, receives a cut out instruction from the user, creates an edited VR content having a second video range that is a video range narrower than the first video range in the VR content, controls to display a second screen that displays, in one screen, a range in the VR content wider than the partial range and including the entirety of the second video range such that the boundary of the second video range can be identified.SELECTED DRAWING: Figure 7

Description

The present invention relates to an electronic device, a method for controlling the electronic device, a program, and a storage medium, and particularly to a control method for displaying an image having a wide range of images.

2. Description of the Related Art In recent years, image capturing apparatuses capable of capturing images having an image in a range wider than a human viewing angle, such as omnidirectional images and omnidirectional images, have become popular. Also, by displaying a part of the image having such a wide range of images on the display and changing the range of the image displayed on the display (display range) in accordance with the change in the posture of the device, the immersive feeling and A method of performing a highly realistic display (VR view) is also known.

When photographing an omnidirectional image, it is often the case that an unwanted subject such as the photographer himself is captured. Therefore, it is useful for an electronic device capable of reproducing an omnidirectional image that the user provides a function of cutting out only a necessary portion or excluding an unnecessary portion.

Patent Documents 1 and 2 disclose a technique in which a user specifies an arbitrary position of an omnidirectional image and cuts out a range including the arbitrary position from the omnidirectional image.

JP, 2013-127739, A JP2012-29180A

Although Patent Documents 1 and 2 disclose a method of cutting out a normal image that is not VR content from an omnidirectional image (VR content), a VR content in a narrower range is extracted from the omnidirectional image (VR content). No consideration is given to cutting out. When cropping a VR content in a narrower range from the VR content, the range of the VR content image after the cropping is narrower than that before the cropping, but the range is larger than the display range displayed at one time in the normal VR view. Is also wide. Therefore, in the operation process of cutting out a VR content in a narrower range from the VR content, it is difficult to confirm which range of the VR content before cutting is included in the range of the video of the VR content after cutting.

Therefore, it is an object of the present invention to provide a technique that makes it possible to more easily confirm a video range of a VR content after clipping when generating a VR content in a narrower range from the VR content.

In order to solve the above problems, the electronic device of the present invention,
On the first screen, control is performed so that a part of the VR content having the first video range is displayed as a display range on the display unit, and the attitude of the electronic device is changed or the display range is changed by the user. Display control means for changing the position of the display range,
A receiving means for receiving a cutout instruction from the user,
Generating means for generating an edited VR content having a second video range which is a video range narrower than the first video range of the VR content;
A second range, which is wider than the partial range of the VR content and includes the entire second video range, is displayed on one screen so that the range of the second video range can be identified. And a control means for controlling to display a screen.

According to the present invention, when a VR content in a narrower range is generated from the VR content, it is possible to more easily confirm the video range of the VR content after cutout.

It is an external view and a block diagram of a digital camera. It is an external view or a block diagram of the electronic device. It is a flow chart of VR view display. It is a flow chart of cutout edit processing. It is a flowchart of a cutout preview process. It is an example of a VR view display, a cutout edit screen, and a preview screen. It is an example of a whole hemisphere display. It is a flowchart of a two-screen display process. It is an example of a two-screen display. It is a flow chart of edge fine adjustment processing. It is an example of a fine edge adjustment screen. It is a flowchart of an unnecessary part designation process. It is an example of a screen for designating unnecessary portions. It is a schematic diagram which shows an example of the flow until a selection invalid range is set. It is an example of a cutout edit screen and a preview screen. It is a schematic diagram which shows an example of the difference of the cutout by the effective/ineffective of an elevation/depression angle. It is an example of guidance display of a cutout range. It is an example of various displays of a VR image.

Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. FIG. 1A is a front perspective view (outside view) of the digital camera 100 (imaging device). FIG. 1B is a rear perspective view (outside view) of the digital camera 100. The digital camera 100 is an omnidirectional camera (omnidirectional camera).

The barrier 102a is a protective window for the front camera unit having a shooting range in front of the digital camera 100. The front camera unit is, for example, a wide-angle camera unit having a wide range of 180 degrees or more on the front side of the digital camera 100 as an imaging range. The barrier 102b is a protective window for the rear camera section, which has a shooting range behind the digital camera 100. The rear camera unit is, for example, a wide-angle camera unit on the rear side of the digital camera 100, which has a wide range of 180 degrees or more in all directions.

The display unit 28 displays various information. The shutter button 61 is an operation unit (operation member) for issuing a shooting instruction. The mode changeover switch 60 is an operation unit for changing over various modes. The connection I/F 25 is a connector for connecting a connection cable to the digital camera 100, and an external device such as a smartphone, a personal computer, or a television is connected to the digital camera 100 using the connection cable. The operation unit 70 is various switches, buttons, dials, touch sensors, etc. that receive various operations from the user. The power switch 72 is a push button for switching the power on/off.

The light emitting unit 21 is a light emitting member such as a light emitting diode (LED), and notifies the user of various states of the digital camera 100 by a light emitting pattern and a light emitting color. The fixing unit 40 is, for example, a tripod screw hole, and is used to fix and install the digital camera 100 with a fixing device such as a tripod.

FIG. 1C is a block diagram showing a configuration example of the digital camera 100.

The barrier 102a covers the image pickup system of the front camera unit (the taking lens 103a, the shutter 101a, the image pickup unit 22a, etc.) to prevent the image pickup system from being soiled or damaged. The taking lens 103a is a lens group including a zoom lens and a focus lens, and is a wide-angle lens. The shutter 101a is a shutter having a diaphragm function for adjusting the amount of subject light incident on the imaging unit 22a. The imaging unit 22a is an imaging device (imaging sensor) configured by a CCD, a CMOS device, or the like that converts an optical image into an electric signal. The A/D converter 23a converts the analog signal output from the imaging unit 22a into a digital signal. Note that the outer surface of the taking lens 103a may be exposed without providing the barrier 102a, and the taking lens 103a may prevent other imaging systems (the shutter 101a and the imaging unit 22a) from being soiled or damaged.

The barrier 102b covers the image pickup system of the rear camera section (the taking lens 103b, the shutter 101b, the image pickup section 22b, etc.) to prevent the image pickup system from being soiled or damaged. The taking lens 103b is a lens group including a zoom lens and a focus lens, and is a wide-angle lens. The shutter 101b is a shutter having a diaphragm function that adjusts the amount of subject light incident on the imaging unit 22b. The image pickup unit 22b is an image pickup device including a CCD, a CMOS device, or the like that converts an optical image into an electric signal. The A/D converter 23b converts the analog signal output from the imaging unit 22b into a digital signal. Note that the outer surface of the taking lens 103b may be exposed without providing the barrier 102b, and the taking lens 103b may prevent other imaging systems (the shutter 101b and the imaging unit 22b) from being soiled or damaged.

A VR (Virtual Reality) image is captured by the image capturing unit 22a and the image capturing unit 22b. The VR image is an image that can be displayed in VR (displayed in the display mode “VR view”). The VR image is an omnidirectional image (omnidirectional image) captured by an omnidirectional camera (omnidirectional camera) or a panoramic image having a wider image range (effective image range) than the display range that can be displayed on the display unit at one time. Etc. shall be included. The VR image includes not only a still image but also a moving image and a live view image (image obtained from a camera in almost real time). The VR image has a maximum visual range of 360 degrees in the vertical direction (vertical angle, angle from zenith, elevation angle, depression angle, altitude angle, pitch angle) and 360 degrees in the horizontal direction (horizontal angle, azimuth angle, yaw angle). Has (effective video range).

Further, even if the VR image is less than 360 degrees up and down and less than 360 degrees left and right, it can be displayed on a wide range of view angles (viewing range) wider than the view angle that can be captured by a normal camera, or at once on the display unit. Images with a wider video range (effective video range) than the display range are also included. For example, an image taken with a omnidirectional camera capable of shooting a subject in a horizontal direction (horizontal angle, azimuth angle) of 360 degrees and a vertical field of view of 210 degrees (angle of view) centered on the zenith is It is a type of VR image. Further, for example, an image captured by a camera capable of capturing a subject in a horizontal direction (horizontal angle, azimuth angle) of 180 degrees and a vertical field of 180 degrees centering on the horizontal direction (angle of view) is a VR image. It is a kind. That is, a VR image is an image that has a visual range of 160 degrees (±80 degrees) or more in each of the vertical direction and the horizontal direction, and that has a wider visual range than a range that a human can visually recognize at one time. Is a kind of.

When this VR image is VR-displayed (displayed in the display mode “VR view”), the posture of the display device (display device displaying the VR image) is changed in the left-right rotation direction, so that the left-right direction (horizontal rotation direction) is changed. You can watch seamless and omnidirectional video. In the vertical direction (vertical rotation direction), a seamless omnidirectional image can be viewed within ±105 degrees from directly above (zenith), but there is no image above 105 degrees from directly above. It becomes a blank area. The VR image can also be said to be an “image in which the video range is at least a part of the virtual space (VR space)”.

The VR display (VR view) is a display method (display mode) capable of changing the display range, in which a video image in a visual field range corresponding to the attitude of the display device is displayed in the VR image. When a head mounted display (HMD), which is a display device, is mounted and viewed, an image in a visual field range corresponding to the orientation of the user's face is displayed. For example, in a VR image, an image with a viewing angle (angle of view) centered at 0° in a horizontal direction (a specific azimuth, for example, north) and 90° in a vertical direction (90° from the zenith, that is, horizontal) at a certain point in time. Is displayed. From this state, when the posture of the display device is reversed upside down (for example, when the display surface is changed from southward to northward), 180 degrees (reverse azimuth, for example, south) of the same VR image is rotated vertically. The display range is changed to an image with a viewing angle centered at 90 degrees (horizontal) in the direction. When the user is watching the HMD, if the user turns their face from north to south (that is, if they face back), the image displayed on the HMD will change from the image in the north to the image in the south. is there. With such a VR display, it is possible to provide the user with a visual feeling as if he/she were in the VR image (VR space). A smartphone attached to VR goggles (head mount adapter) can be said to be a type of HMD.

The display method of the VR image is not limited to the above. The display range may be moved (scrolled) according to a user operation on the touch panel, direction buttons, or the like, instead of the posture change. In VR display (in the display mode “VR view”), in addition to changing the display range by changing the posture, it is possible to change the display range according to touch move on the touch panel, drag operation on the mouse, etc., pressing direction buttons, etc. You may do it.

The image processing unit 24 performs resizing processing such as predetermined pixel interpolation and reduction and color conversion processing on the data from the A/D converter 23a and the A/D converter 23b or the data from the memory control unit 15. The image processing unit 24 also performs a predetermined arithmetic process using the captured image data. The system control unit 50 performs exposure control and distance measurement control based on the calculation result obtained by the image processing unit 24. As a result, TTL (through-the-lens) AF (auto focus) processing, AE (auto exposure) processing, EF (flash pre-emission) processing, and the like are performed. The image processing unit 24 further performs predetermined arithmetic processing using the captured image data, and performs TTL AWB (auto white balance) processing based on the obtained arithmetic result. In addition, the image processing unit 24 performs basic image processing on the two images (two fish-eye images; two wide-angle images) obtained from the A/D converter 23a and the A/D converter 23b. A single VR image is generated by performing joint image processing for synthesizing two images that have been subjected to the conventional image processing. In addition, the image processing unit 24 performs image cutting processing, enlargement processing, distortion correction, etc. for VR display of a VR image during VR display in live view or during playback, and draws the processing result in the VRAM of the memory 32. Render.

In the stitched image processing, the image processing unit 24 uses one of the two images as a reference image and the other as a comparison image, calculates the amount of deviation between the reference image and the comparison image for each area by pattern matching processing, and calculates the shift amount for each area. The connection position that connects the two images is detected based on the shift amount. The image processing unit 24 corrects the distortion of each image by geometric conversion in consideration of the detected connection position and the lens characteristics of each optical system, and converts each image into a spherical image format (a spherical image format). Convert to image. Then, the image processing unit 24 generates one spherical image (VR image) by synthesizing (blending) two spherical images. The generated celestial sphere image is an image using, for example, the equirectangular projection, and the position of each pixel of the celestial sphere image can be associated with the coordinates of the surface of the sphere (VR space).

The output data from the A/D converters 23a and 23b is written in the memory 32 via the image processing unit 24 and the memory control unit 15, or via the memory control unit 15 without the image processing unit 24. The memory 32 stores image data obtained by the imaging units 22a and 22b and converted into digital data by the A/D converters 23a and 23b, and image data to be output from the connection I/F 25 to an external display. The memory 32 has a storage capacity sufficient to store a predetermined number of still images, a moving image for a predetermined time, and audio.

The memory 32 also serves as a memory (video memory) for displaying an image. The image display data stored in the memory 32 can be output from the connection I/F 25 to an external display. Functions as an electronic viewfinder by sequentially transferring and displaying the VR images captured by the image capturing units 22a and 22b and generated by the image processing unit 24 and stored in the memory 32 to an external display. Can be realized, and live view display (LV display) can be performed. Hereinafter, the image displayed in live view display will be referred to as a live view image (LV image). In addition, the VR image stored in the memory 32 is transferred to an external device (smartphone or the like) wirelessly connected via the communication unit 54 and is displayed on the external device side so that live view display (remote LV display) can be performed. ..

The non-volatile memory 56 is a memory as a recording medium that can be electrically erased and recorded, and is, for example, an EEPROM or the like. In the non-volatile memory 56, constants for operating the system control unit 50, programs, etc. are recorded. The program mentioned here is a computer program for executing various processes.

The system control unit 50 is a control unit having at least one processor or circuit and controls the entire digital camera 100. The system control unit 50 realizes each process by executing the program recorded in the nonvolatile memory 56 described above. The system memory 52 is, for example, a RAM. In the system memory 52, constants and variables for operation of the system control unit 50, programs read from the non-volatile memory 56, and the like are expanded. The system control unit 50 also controls display by controlling the memory 32, the image processing unit 24, the memory control unit 15, and the like. The system timer 53 is a time measuring unit that measures time used for various controls and time of a built-in clock.

The mode switch 60, the shutter button 61, the operation unit 70, and the power switch 72 are used to input various operation instructions to the system control unit 50.

The mode switch 60 switches the operation mode of the system control unit 50 to one of a still image recording mode, a moving image shooting mode, a reproduction mode, a communication connection mode, and the like. The modes included in the still image recording mode include an auto shooting mode, an auto scene determination mode, a manual mode, an aperture priority mode (Av mode), a shutter speed priority mode (Tv mode), and a program AE mode. In addition, there are various scene modes, custom modes, and the like, which are shooting settings for each shooting scene. The mode switch 60 allows the user to directly switch to one of these modes. Alternatively, after temporarily switching to the shooting mode list screen with the mode switching switch 60, it may be possible to selectively switch to one of the plurality of modes displayed on the display unit 28 by using another operation member. Similarly, the moving image shooting mode may include a plurality of modes.

The shutter button 61 includes a first shutter switch 62 and a second shutter switch 64. The first shutter switch 62 is turned on during operation of the shutter button 61 by so-called half-pressing (shooting preparation instruction) to generate the first shutter switch signal SW1. The system control unit 50 starts shooting preparation operations such as AF (auto focus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, and EF (flash pre-flash) processing by the first shutter switch signal SW1. To do. The second shutter switch 64 is turned on when the operation of the shutter button 61 is completed, that is, when the shutter button 61 is fully pressed (imaging instruction), and generates the second shutter switch signal SW2. By the second shutter switch signal SW2, the system control unit 50 starts a series of photographing processing operations from signal reading from the image pickup units 22a and 22b to writing image data on the recording medium 90.

Note that the shutter button 61 is not limited to an operation member that can be operated in two steps of full-press and half-press, and may be an operation member that can be pressed in only one step. In that case, the photographing preparation operation and the photographing process are continuously performed by one-step depression. This is the same operation as when the shutter button that can be half-pressed and full-pressed is fully pressed (when the first shutter switch signal SW1 and the second shutter switch signal SW2 are generated almost at the same time).

The operation unit 70 is appropriately assigned a function for each scene by selecting and operating various function icons and options displayed on the display unit 28, and functions as various function buttons. The function buttons include, for example, an end button, a return button, an image advance button, a jump button, a narrow-down button, an attribute change button and the like. For example, when the menu button is pressed, various settable menu screens are displayed on the display unit 28. The user can intuitively perform various settings by operating the operation unit 70 while looking at the menu screen displayed on the display unit 28.

The power switch 72 is a push button for switching the power on/off. The power supply control unit 80 includes a battery detection circuit, a DC-DC converter, a switch circuit that switches blocks to be energized, and the like, and detects whether or not a battery is mounted, the type of battery, the remaining battery level, and the like. Further, the power supply control unit 80 controls the DC-DC converter based on the detection result and the instruction of the system control unit 50, and supplies a necessary voltage to each unit including the recording medium 90 for a necessary period. The power supply unit 30 includes a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, an AC adapter, or the like.

The recording medium I/F 18 is an interface with a recording medium 90 such as a memory card or a hard disk. The recording medium 90 is a recording medium such as a memory card for recording captured images, and is composed of a semiconductor memory, an optical disc, a magnetic disc, or the like. The recording medium 90 may be an exchange recording medium that can be attached to and detached from the digital camera 100, or may be a recording medium built in the digital camera 100.

The communication unit 54 transmits/receives a video signal, an audio signal, etc. to/from an external device connected by a wireless or wired cable. The communication unit 54 is also connectable to a wireless LAN (Local Area Network) and the Internet. The communication unit 54 can transmit images (including LV images) captured by the image capturing units 22a and 22b and images recorded on the recording medium 90, and can receive images and various other information from external devices. ..

The posture detection unit 55 detects the posture of the digital camera 100 with respect to the direction of gravity. Based on the posture detected by the posture detection unit 55, the images photographed by the image pickup units 22a and 22b are images photographed with the digital camera 100 held horizontally or vertically. It is possible to determine whether there is. Further, it is possible to determine how much the images captured by the image capturing units 22a and 22b are images captured with the digital camera 100 tilted in the three axial directions (rotational directions) of the yaw direction, the pitch direction, and the roll direction. is there. The system control unit 50 adds orientation information according to the orientation detected by the orientation detection unit 55 to the image file of the VR image captured by the image capturing units 22a and 22b, and rotates the image (tilt correction (zenith correction) It is possible to adjust the orientation of the image so that it is recorded. One sensor or a combination of a plurality of sensors such as an acceleration sensor, a gyro sensor, a geomagnetic sensor, a direction sensor, and an altitude sensor can be used as the attitude detection unit 55. It is also possible to detect the movement of the digital camera 100 (panning, tilting, lifting, whether it is stationary, etc.) by using an acceleration sensor, a gyro sensor, a direction sensor, and the like that form the attitude detection unit 55.

The microphone 20 is a microphone that collects sound around the digital camera 100, which is recorded as sound of a VR image (VR moving image) that is a moving image. The connection I/F 25 is a connection plug to which an HDMI (registered trademark) cable, a USB cable, or the like is connected so as to connect to an external device and transmit/receive video.

FIG. 2A is an external view of an electronic device 200 that is a type of display control device. The electronic device 200 is a display device such as a smartphone. The display 205 is a display unit that displays images and various information. The display 205 is configured integrally with the touch panel 206a, and can detect a touch operation on the display surface of the display 205. The electronic device 200 can display the VR image (VR content) in the VR on the display 205. The operation unit 206b is a power button that receives an operation for switching the power of the electronic device 200 on and off. The operation unit 206c and the operation unit 206d are volume buttons for increasing/decreasing the volume of sound output from the speaker 212b, an earphone connected to the sound output terminal 212a, an external speaker, or the like. The operation unit 206e is a home button for displaying a home screen on the display 205. The audio output terminal 212a is an earphone jack and is a terminal that outputs an audio signal to an earphone, an external speaker, or the like. The speaker 212b is a main body built-in speaker that outputs sound.

FIG. 2B is a block diagram showing a configuration example of the electronic device 200. The CPU 201, the memory 202, the non-volatile memory 203, the image processing unit 204, the display 205, the operation unit 206, the recording medium I/F 207, the external I/F 209, and the communication I/F 210 are connected to the internal bus 250. .. The voice output unit 212 and the posture detection unit 213 are also connected to the internal bus 250. The respective units connected to the internal bus 250 are capable of exchanging data with each other via the internal bus 250.

The CPU 201 is a control unit that controls the entire electronic device 200, and includes at least one processor or circuit. The memory 202 is, for example, a RAM (a volatile memory using a semiconductor element or the like). The CPU 201 controls each unit of the electronic device 200 by using the memory 202 as a work memory according to a program stored in the non-volatile memory 203, for example. The non-volatile memory 203 stores image data, audio data, other data, various programs for operating the CPU 201, and the like. The non-volatile memory 203 is composed of, for example, a flash memory or a ROM.

The image processing unit 204, under the control of the CPU 201, an image stored in the nonvolatile memory 203 or the recording medium 208, a video signal acquired via the external I/F 209, or an image acquired via the communication I/F 210. Various image processing is performed on the above. The image processing performed by the image processing unit 204 includes A/D conversion processing, D/A conversion processing, image data encoding processing, compression processing, decoding processing, enlargement/reduction processing (resizing), noise reduction processing, and color conversion. Processing etc. are included. In addition, various image processing such as panoramic expansion, mapping processing, and conversion of a VR image that is a wide-range image having a wide-range image even if not an omni-directional image or an omni-directional image is performed. The image processing unit 204 may be composed of a dedicated circuit block for performing specific image processing. In addition, depending on the type of image processing, the CPU 201 may perform image processing according to a program without using the image processing unit 204.

The display 205 displays an image or a GUI screen forming a GUI (Graphical User Interface) based on the control of the CPU 201. The CPU 201 controls each unit of the electronic device 200 so as to generate a display control signal according to a program, generate a video signal for displaying on the display 205, and output the video signal to the display 205. The display 205 displays an image based on the generated and output image signal. The electronic device 200 itself may include an interface for outputting a video signal to be displayed on the display 205, and the display 205 may be an external monitor (TV, HMD, etc.).

The operation unit 206 is an input device for receiving a user operation, which includes a character information input device such as a keyboard, a pointing device such as a mouse or a touch panel, a button, a dial, a joystick, a touch sensor, and a touch pad. In this embodiment, the operation unit 206 includes a touch panel 206a and operation units 206b, 206c, 206d, and 206e.

A recording medium 208 such as a memory card, a CD or a DVD can be attached to and detached from the recording medium I/F 207. The recording medium I/F 207 reads data from the attached recording medium 208 and writes data to the recording medium 208 under the control of the CPU 201. The recording medium 208 stores data such as images to be displayed on the display 205. The external I/F 209 is an interface for connecting to an external device by a wired cable (USB cable or the like) or wirelessly and performing input/output (data communication) of a video signal or an audio signal. The communication I/F 210 is an interface for communicating (wireless communication) with an external device, the Internet 211, or the like to transmit and receive various data such as files and commands (data communication).

The voice output unit 212 outputs voices of moving images and music data reproduced by the electronic device 200, operation sounds, ring tones, various notification sounds, and the like. The audio output unit 212 includes an audio output terminal 212a for connecting an earphone and the like and a speaker 212b, but the audio output unit 212 may output audio data to an external speaker by wireless communication or the like.

The posture detection unit 213 detects the posture (tilt) of the electronic device 200 with respect to the gravity direction and the posture of the electronic device 200 with respect to each axis in the yaw direction, the pitch direction, and the roll direction, and notifies the CPU 201 of the posture information. Based on the posture detected by the posture detection unit 213, whether the electronic device 200 is held horizontally, held vertically, oriented upward, oriented downward, or in an inclined posture, etc. Can be determined. Further, it is possible to determine whether or not the electronic device 200 is tilted in the rotation direction such as the yaw direction, the pitch direction, and the roll direction, and the size thereof, and whether the electronic device 200 is rotated in the rotation direction. One sensor or a combination of a plurality of sensors such as an acceleration sensor, a gyro sensor, a geomagnetic sensor, a direction sensor, and an altitude sensor can be used as the attitude detection unit 213.

As described above, the operation unit 206 includes the touch panel 206a. The touch panel 206a is an input device that is configured to be superposed on the display 205 and has a planar shape, and outputs coordinate information corresponding to the touched position. The CPU 201 can detect the following operation on the touch panel 206a or the state thereof.
-A finger or pen that has not touched the touch panel 206a newly touches the touch panel 206a, that is, the start of touch (hereinafter, referred to as Touch-Down).
A state where a finger or a pen is touching the touch panel 206a (hereinafter, referred to as touch-on).
-A finger or pen moving while touching the touch panel 206a (hereinafter referred to as Touch-Move)
The finger or pen touching the touch panel 206a is separated from the touch panel 206a, that is, the touch ends (hereinafter, referred to as Touch-Up).
-A state in which nothing is touched on the touch panel 206a (hereinafter referred to as touch-off)

When touch down is detected, touch on is also detected. After touchdown, touchon is normally detected unless touchup is detected. Even if a touch move is detected, touch-on is detected at the same time. Even if the touch-on is detected, the touch move is not detected unless the touch position is moved. When it is detected that all the touched fingers or pens are touched up, touch-off is detected.

These operations/states and the position coordinates of the finger or pen touching the touch panel 206a are notified to the CPU 201 through the internal bus, and the CPU 201 performs any operation (touch operation) on the touch panel 206a based on the notified information. ) Is performed. With respect to the touch move, the moving direction of the finger or the pen moving on the touch panel 206a can be determined for each vertical component/horizontal component on the touch panel 206a based on the change in the position coordinates. When it is detected that the touch move is performed over the predetermined distance, it is determined that the slide operation is performed. The touch move is a moving operation performed by the user on the touch panel 206a. Various processes according to a touch move described later can also be performed according to a drag operation with a mouse, which is also a moving operation.

An operation of quickly moving a certain distance while touching the finger on the touch panel 206a and releasing the finger is called a flick. In other words, the flick is an operation of quickly tracing the touch panel 206a with a finger. When it is detected that a touch move is performed over a predetermined distance or more at a predetermined speed or more, and a touch-up is detected as it is, it can be determined that a flick has been performed (it can be determined that a flick follows a slide operation).

Further, a touch operation of simultaneously touching a plurality of points (for example, two points) to bring the touch positions closer to each other is called a pinch-in, and a touch operation of moving the touch positions away from each other is called a pinch-out. Pinch out and pinch in are collectively referred to as a pinch operation (or simply pinch). The touch panel 206a may be any of various types of touch panels such as a resistive film type, a capacitance type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, and an optical sensor type. Good. There is a method of detecting that there is a touch due to the touch on the touch panel, and a method of detecting that there is a touch due to the approach of the finger or pen to the touch panel, but any method may be used.

FIG. 2C is an external view of a VR goggle (head mount adapter) 230 to which the electronic device 200 can be attached. The electronic device 200 can also be used as a head mounted display by mounting it on the VR goggles 230. The insertion port 231 is an insertion port for inserting the electronic device 200. The entire electronic device 200 can be inserted into the VR goggles 230 with the display surface of the display 205 facing the headband 232 side (that is, the user side) for fixing the VR goggles 230 to the user's head. The user can view the display 205 while holding the VR goggles 230 to which the electronic device 200 is attached on his/her head without holding the electronic device 200 by hand. In this case, when the user moves the head or the whole body, the posture of the electronic device 200 also changes. The attitude detection unit 213 detects the attitude change of the electronic device 200 at this time, and the CPU 201 performs a process for VR display based on the attitude change. In this case, detecting the posture of the electronic device 200 by the posture detection unit 213 is equivalent to detecting the posture of the user's head (the direction in which the user's line of sight is facing). The electronic device 200 itself may be an HMD that can reach the head without the VR goggles.

In the present embodiment, a VR image having a video range (effective video range) of 360 degrees in the vertical and horizontal directions at the maximum, and a VR image having a video range of 180 degrees in the vertical and horizontal directions (effective video range). The cutting operation and processing will be described. Note that, hereinafter, a VR image having a video range (effective video range) of 360 degrees in the vertical and horizontal directions at the maximum is referred to as a 360-degree VR image. A VR image having a video range (effective video range) of 180 degrees of visual field in the vertical and horizontal directions is referred to as a 180-degree VR image. The process of cutting out a 180-degree VR image from a 360-degree VR image is conceptually a process of cutting out a hemisphere from a virtual sphere onto which a 360-degree video image is mapped. The cutout described in the present embodiment is not a process of cutting out a normal rectangular image (planar image), but a process of cutting out an image that can be viewed as a VR image even after cutout (that can be viewed in a VR view by mapping on a sphere). Is. Since the image after cutting has an effective image range of 180 degrees in the vertical and horizontal directions, when it is mapped on a sphere, an effective image of a hemisphere (a captured image or the like) is mapped. The remaining hemisphere corresponds to the invalid video range, and the image filled with a single color or a predetermined pattern, or a video that is complemented in some way is mapped. When viewing such a VR image after cutout in the VR view, the user can visually recognize the image in a front width range of 180 degrees (90 degrees from the center in the vertical and horizontal directions). The reason for performing such a cutting process is as follows.

First, the amount of image data can be reduced. The cut-out 180-degree VR image has a smaller amount of data than the 360-degree VR image. Therefore, it is possible to suppress the capacity pressure of the recording medium during storage. In addition, it is possible to reduce the amount of communication data when transmitting and receiving, and to reduce the processing load when displaying, and accordingly, there are effects such as improvement in processing speed and response speed, and reduction in power consumption.

Secondly, unnecessary subjects can be deleted. For example, in most cases of capturing a VR image having an effective image range for a field of view of 360 degrees, it is inevitable that the photographer himself/herself will be reflected. However, when the photographer wants to photograph a landscape instead of the photographer himself, the photographer himself is an unnecessary subject and is not intended. When an unintended image is captured, it prevents the viewer from concentrating the viewer's attention on the image such as a landscape that the photographer thought was the subject. In addition, confidential information such as the face of a passerby or the license plate of a passing vehicle is often captured, which may be undesirable from the viewpoint of privacy and security. These problems can be avoided by cutting out only the range intended by the photographer and discarding the images in other unnecessary ranges by the cutout process.

Thirdly, it is possible to reduce the physical burden on the user who views the VR view. When viewing a VR image of 360 degrees in a VR view, an image exists behind the viewing user. If the image is for 180 degrees ahead, the viewer can look around by simply changing the direction of his/her neck, but if he wants to look behind him, he must twist his/her upper body or change the standing direction. However, the physical burden is rather high. Such a physical movement to look behind is not suitable for sitting on a chair for viewing unless the chair is a swivel chair. On the other hand, in a VR image of 180 degrees, the entire body can be overlooked simply by changing the direction of one's own neck, so that the physical burden on viewing is relatively small. If the viewer recognizes that the VR image is a 180-degree VR image by a guide display or a display that shows that the range exceeding 180 degrees is an invalid area, the viewer may try to look behind. Absent. Therefore, it is possible to prevent the viewer from performing a physical action such as looking back.

Fourth, it is possible to display the VR image of 360 degrees in correspondence with the recording format for the VR image of 180 degrees. When the recording format of a 360-degree VR image is different from the recording format of a 180-degree VR image, a playback device (playback application software) that only supports viewing of a 180-degree VR image can output a 360-degree VR image. You cannot view the image. However, if a 180-degree VR image is generated by cutting out a 360-degree VR image, it can be viewed even by a playback device that is only capable of viewing the 180-degree VR image. Further, even if both the 180-degree VR image viewing and the 360-degree VR image viewing are supported, it is necessary to switch between the 180-degree VR image viewing mode and the 360-degree VR image viewing mode. In some cases, the switching operation is troublesome. However, if a 180-degree VR image is generated by cutting out from a 360-degree VR image, it is possible to sequentially switch and display a plurality of VR images without switching the display mode for the 180-degree VR image. In addition, two 180-degree VR images for the right eye and the left eye, respectively, captured by two camera units installed at a distance apart by parallax toward the subject in the same direction, are displayed so that they can be stereoscopically viewed. A display mode (binocular VR view) is assumed. If a 180-degree VR image for the right eye and a 180-degree VR image for the left eye are cut out and recorded from the 360-degree VR image, it is possible to view even such a two-view VR view.

In the present embodiment, an example in which a 180-degree VR image is cut out and generated from a 360-degree VR image will be described. However, a VR image having an effective video range narrower than the effective video range of the VR image before cutting is cut out. The processing can be applied to VR images with other viewing angles. That is, the VR image before cutout is not limited to the VR image of 360 degrees, and the VR image after cutout is not limited to the VR image of 180 degrees.

In each process described below, an example will be described in which various touch buttons are displayed and a touch operation on each touch button is received as a user operation to proceed with the process. Specifically, the touch operation of receiving instructions for various touch buttons may be a touch-up from a touch reaction area corresponding to the touch button or a touch-down to a touch reaction area corresponding to the touch button. Moreover, the present invention is not limited to this, and each instruction described as being accepted by an operation on a touch button described later may be accepted by another user operation. For example, an operation to each physical button, an operation to select a display item using the direction keys and press the enter button, an operation to select a display item using the mouse, a voice command, or the like may be used.

FIG. 3 is a flowchart of VR view display (processing) in the electronic device 200. This processing is realized by loading the program recorded in the non-volatile memory 203 into the memory 202 and executing it by the CPU 201. When the electronic device 200 is turned on, a VR image of 360 degrees can be displayed in a VR view, and when application software having a function of cutting out a VR image of 180 degrees is started, the CPU 201 initializes flags and control variables. After conversion, the processing of FIG. 3 is started.

In S301, the CPU 201 reads (acquires) the VR image to be displayed from the communication destination via the recording medium 208 or the communication I/F 210. Further, information indicating a video range (effective video range), which is incidental to the VR image attribute information, is acquired. The information indicating the effective image range is information indicating which angle range of the effective image the VR image has in the vertical direction and the horizontal direction. The information indicating the effective video range may be information that can specify the effective video range of the VR image, and may be information such as the angle of view, the viewing angle, the azimuth angle, the elevation angle, the number of left and right pixels, and the coordinates. Further, the information indicating the effective video range may be model information of the camera that captured the VR image (the range that can be photographed can be specified by specifying the model), zoom information at the time of photographing, or the like. When the CPU 201 acquires the information indicating the effective video range, the CPU 201 also specifies (calculates) the invalid video range (non-video range) based on the difference between the upper, lower, left, and right 360 degrees. On the contrary, the information indicating the invalid video range may be acquired from the attribute information of the VR image, and the valid video range may be calculated from the information, or both the information indicating the valid video range and the information indicating the invalid video range may be obtained. It may be acquired from the attribute information of the VR image.

In S302, the CPU 201 performs display processing in the VR view of the VR image acquired in S301. The original data (image data) of the VR image is, for example, a distorted image using the equirectangular projection, and is in an image format in which the position of each pixel can be associated with the coordinates of the surface of the sphere. The original image of this VR image is mapped on a sphere, and a part of it is cut out and displayed. That is, the image displayed in S302 is an image obtained by cutting out and enlarging a part of the VR image, and is an image in which the distortion of the original image is removed (or reduced).

In S303, the CPU 201 determines, based on the information acquired in S301, whether the VR image to be displayed in S302 is a VR image capable of cropping a VR image of 180 degrees. In the case where the number of recorded pixels is less than the threshold value, the effective video range is narrower (smaller) than the threshold value, the image which has already been subjected to the clipping process described later, or the 180-degree VR image, It is determined that the VR image is not a croppable VR image, and the process proceeds to S305. When it is determined that the VR image can be cut out, the process proceeds to S304.

In S304, the CPU 201 displays a cutout edit button. FIG. 6A shows a display example of the VR image and the cutout edit button displayed in S304. The VR image 602 is the VR image displayed in S302. The cutout edit button 601 is a touch icon for instructing cutout of a VR image. If it is determined in step S303 that the VR image is not a croppable VR image, the cropping edit button 601 is not displayed, but the cropping is performed in a display form such as grayout that can be identified as being in an unselectable state. The edit button 601 may be displayed.

In S305, the CPU 201 determines whether or not there is a touch move (display range changing operation) to the area in which the VR image 602 is displayed, based on the information notified from the touch panel 206a. If there is a touch move, the process proceeds to S306, and if not, the process proceeds to S307. In step S306, the CPU 201 scrolls the displayed VR image 602 according to the touch-moved direction and amount, and changes the display range of the VR image. Specifically, the display range is rotationally moved around the vertical axis (zenith axis, gravity axis) of the virtual sphere onto which the VR image is mapped, based on the horizontal (horizontal) component of the movement component of the touch move. As described above, the VR image is scrolled rightward or leftward. That is, the display range is changed by changing the azimuth of the line of sight (direction of the display range) viewed from the viewer placed at the center of the virtual sphere. In addition, based on the vertical (vertical) component of the movement component of the touch move, the horizontal axis (vertical axis to the zenith axis, vertical axis to gravity) of the virtual sphere onto which the VR image is mapped is displayed as the center. Scroll the VR image up or down so that the range rotates. That is, the display range is changed by changing the elevation angle or depression angle (hereinafter referred to as elevation angle) of the line of sight (direction of the display range) from the viewer placed at the center of the virtual sphere.

In step S<b>307, the CPU 201 determines whether the posture detection unit 213 has detected a change in the posture in which the display range should be changed. If a change in the posture for changing the display range is detected, the process proceeds to S308, and if not, the process proceeds to S309. In S308, the CPU 201 changes (moves) the display range of the VR image 602 according to the posture change detected by the posture detection unit 213.

In S309, the CPU 201 determines, based on the information notified from the touch panel 206a, whether or not there is a pinch-in or a pinch-out (that is, a magnification change operation) with respect to the area in which the VR image 602 is displayed. If there is pinch-in or pinch-out, the process proceeds to S310, and if not, the process proceeds to S311. In S310, the CPU 201 changes the display magnification of the VR image (continuously or stepwise in three or more stages) in response to the pinch operation. In the case of pinch-in (reduction operation), the display magnification of the VR image is lowered. Along with this, the display range of the VR image becomes wider. That is, the subject is smaller, but a wider range is displayed. In the case of pinch out (enlargement operation), increase the display magnification of the VR image. Along with this, the display range of the VR image becomes narrower. That is, the subject becomes larger, but a narrower range is displayed.

In S311, the CPU 201 determines, based on the information notified from the touch panel 206a, whether or not there is a touch operation for designating the cutout edit button 601. If the cutout edit button 601 has been touched, the process proceeds to S312, and if not, the process proceeds to S313. In S312, the CPU 201 performs cutout edit processing. The cutout editing process will be described later with reference to FIG.

In S313, the CPU 201 determines whether or not an image switching operation has been performed. If there is an image switching operation, the process proceeds to S314, and if not, the process proceeds to S315. For the image switching operation, for example, an operation of an image switching button included in the operation unit 206, a touch operation of an image switching icon (image forwarding icon, image returning icon) displayed on the display 205, and two points are touched at the same time. Double drag to drag in the same direction. In step S314, the CPU 201 specifies a VR image of an image file different from the currently displayed VR image as the next display target in response to the image switching operation. Subsequently, the process returns to S301, and the VR image to be displayed next is acquired and displayed.

In S315, the CPU 201 determines whether or not another operation has been performed. If there is another operation, the process proceeds to S316, and if not, the process proceeds to S317. In S316, the CPU 201 carries out processing according to other operations. Examples of processing according to other operations include display processing of a VR image in a twin-view VR view, deletion of a file of the displayed VR image, sharing (posting to SNS or sending mail, instruction to use in other applications). Etc.), editing processing such as color adjustment.

In S317, the CPU 201 determines whether or not there is an end operation on the operation unit 206. If there is no end operation, the process returns to S305 and repeats. If there is an end operation, the application software that is the VR image display application is closed, and the processing in FIG. 3 ends.

FIG. 4 shows a flowchart of the cutout editing process described above in S312 of FIG. This processing is realized by loading the program recorded in the non-volatile memory 203 into the memory 202 and executing it by the CPU 201. The editing process of FIG. 4 is basically a process of designating the center of the range cut out as a VR image of 180 degrees out of the VR image of 360 degrees displayed in the VR view as the VR view. Although it is possible to perform fine adjustment for specifying the range, the range to be cut out can be determined only by roughly determining the center without performing fine adjustment, so that the range to be cut out can be easily specified. In the VR view, in many cases, the entire VR image is not displayed, and the boundary of the clipping range does not become a straight line boundary as in the case of the planar image. Therefore, if a method of displaying the trimming frame and determining the trimming range by adjusting the displayed trimming frame is used, it is complicated and difficult for the user to understand. On the other hand, according to the cutout editing process of the present embodiment, the cutout range can be specified only by specifying the center of the cutout range as a 180-degree VR image out of the 360-degree VR image in the VR view. .. For this reason, the user can cut out with an intuitive and easy-to-understand and simple operation feeling.

In step S401, the CPU 201 displays a cutout edit screen, which is a screen for specifying the cutout range, on the display 205. FIG. 6B shows a display example of the cutout edit screen. The VR image 602 is the same VR image as the image displayed in the VR view display before the transition to the cutout edit screen. The display range can be enlarged or reduced by a pinch operation. For example, immediately after the cutout edit screen is displayed (default state), the VR image 602 is displayed in a display range narrower than the range of 180 degrees. The pointer 603 is an indicator for designating the center of the cutout range (reference position of the cutout range). The cutout button 604 is an instruction icon for designating the cutout range based on the current display range (reference position). On the cutout edit screen, the pointer 603 is displayed at a fixed position in the center of the screen, and even if the display range of the VR image 602 changes, the position of the pointer 603 on the display 205 does not change. The user operates the display range of the VR image 602 displayed in the VR view in the normal VR view so that the position of the desired subject, which should be the center of the VR image after clipping, becomes the position indicated by the pointer 603. Change by almost the same operation. Then, by touching the cutout button 604, a range based on the current display range (reference position) can be designated (or provisionally designated) as the range after cutout. More specifically, the range after cutting can be specified so that the position of the VR image corresponding to the position where the pointer 603 is displayed is the center after cutting or the orientation of the center.

In the circular display direction guide, the fan-shaped display range 605 indicates the azimuth range (angle range around the zenith axis) of the current display range in all directions of the VR image shown by the circle. .. The non-display range 606 indicates a range of directions (angle range around the zenith axis) that is not the current display range in all directions of the VR image indicated by a circle. The reference direction 607 is a direction (hereinafter referred to as a display start direction) that is the center of the display range to be displayed first (that is, before the operation of changing the display range and the posture change occurs) when the display of the VR image 602 is started. Showing.

The angle information 608 (angle scale) indicates the current angle (azimuth) of the pointer 603 with respect to the display start direction, and the display is updated when the display range is changed. The whole hemisphere display button 609 is an instruction icon for switching the display magnification of the VR image 602 to a magnification including the entire range specified as the range after cutout with one touch. The two-screen confirmation button 610 is for switching to a display mode in which both a range specified as a cutout range and a deletion range not specified as a cutout range and discarded in a VR image after cutout are displayed at the same time. It is an instruction icon. The edge fine adjustment button 611 is an instruction icon for switching to a display mode in which the edge (boundary portion) of the cutout range is displayed in order to finely adjust the cutout range. The unnecessary portion designation button 612 is an instruction icon for switching to a mode for designating an unnecessary portion that is a subject that should not be included in the cutout range. When the unnecessary portion is designated, when the center of the cutout range is designated so as not to include the unnecessary portion, the designated range and the undesignated range are displayed in a distinguishable manner.

The elevation/depression angle valid/invalid button 613 validates or invalidates the elevation/depression angle (the elevation/depression angle at the position indicated by the pointer 603 in the entire VR image mapped on the sphere) corresponding to the current display range as the center of the cutout range. This is an instruction icon for switching whether or not. Further, the elevation/depression angle valid/invalid button 613 also shows the current setting state, and is displayed as an icon of a display form indicating that the current setting state is valid, and invalid if the current setting state is invalid. Is displayed as an icon in the display form. When enabled, the azimuth and the elevation/depression angle indicated by the pointer 603 are set as the centers, and the surroundings are 90 degrees, respectively, and a total range of 180 degrees (hemisphere range) is set as the cutout range. When disabled, the elevation angle is not based on the elevation/depression angle indicated by the pointer 603, and the elevation/depression angle is 0 degree, that is, a range of 90 degrees around the position indicated by the pointer 603 on the horizontal plane, and a total of 180 degrees (hemisphere range). To the cutout range. In the initial setting before the user performs the specified operation, the setting of valid/invalid elevation/depression is invalid.

The posture detection ON/OFF button 614 is an instruction icon for switching between validating and invalidating the change of the display range of the VR image 602 due to the posture change. Further, the posture detection ON/OFF button 614 also shows the current setting state, and is displayed as an icon of a display form indicating that the current setting state is valid, and invalid if the current setting state is invalid. Is displayed as an icon in the display form. When enabled, the center of the cutout range can be determined simply by changing the posture of the electronic device 200, which is simple and intuitive, while the display range is frequently changed only by changing the posture. It may be difficult to adjust the display range. By disabling it, you can prevent the display range from being changed due to posture changes, prevent the display range from being changed unintentionally, and use the touch move operation (slide operation) to make the display range as intended by the user. Can be adjusted. In the initial setting before the user performs the designation operation, the posture detection ON/OFF setting may be invalidated (OFF).

In S402 and S403, the CPU 201 changes the display range according to the touch move for the area where the VR image 602 is displayed. The details of the processing are the same as S305 and S306 in FIG. 3 described above. As described above, the display position of the pointer 603 on the display 205 does not change even if the display range changes. If the display range of the VR image changes, the relative positional relationship of the pointer 603 with respect to the VR image changes. Further, by this processing, the width of the cutting range (180 degrees) in the cutting processing described later in S502 and S503 does not change, and the position of the cutting range in the VR image before cutting is changed.

FIG. 6C shows a display example when the display range is changed by touch-moving to the left from the state of FIG. 6B. The display range of the VR image 602 is changed, and the angle information 608 and the ranges indicated by the display range 605 and the non-display range 606 of the display direction guide are changed accordingly. The VR image 602 is scrolled so as to move to the left as compared to the state in FIG. 6B, and the right side region of the VR image is newly set as the display range compared to the state in FIG. 6B. .. That is, the display range has moved to the right. In the example of FIG. 6C, as can be seen from the angle information 608 and the display direction guide, the display range is changed in a direction that is oriented approximately 90 degrees to the right as compared with the display range of FIG. 6B. In this display range, a car passing in the direction in which the license plate can be seen is shown as a subject. For example, the user may want to prevent such a license plate of a passing car from being included in the image after being cut out, which is not information to be disclosed.

FIG. 6D shows a display example when the display range is changed by further touch-moving to the left from the state of FIG. 6C. The display range of the VR image 602 is changed, and the angle information 608 and the ranges indicated by the display range 605 and the non-display range 606 of the display direction guide are changed accordingly. The VR image 602 is scrolled so as to move to the left side as compared with FIG. 6C, and the right side region of the VR image is newly set as the display range as compared with the state of FIG. 6C. .. That is, the display range has moved to the right. In the example of FIG. 6D, as can be seen from the angle information 608 and the display direction guide, the display range is changed to a direction substantially right behind (180 degrees to the right) as compared with the display range of FIG. 6B. ing. In this display range, the photographer himself/herself is shown as a subject. For example, the user thinks that such a photographer himself/herself is not the main subject he/she wanted to photograph and does not want to be included in the image after clipping.

In S404, the CPU 201 refers to the setting information stored in the memory 202, and determines whether or not the display range changing function according to the posture detection is valid (ON). Validity (ON)/invalidity (OFF) of the display range changing function according to the posture detection is a setting item set by a touch operation on the posture detection ON/OFF button 614 described above. If the function of changing the display range according to the attitude detection is valid (ON), the process proceeds to S405, and if not, the process proceeds to S407. In S405 and S406, the CPU 201 changes the display range of the VR image 602 according to the change in the posture detected by the posture detection unit 213. This process is the same as S307 and S308 in FIG. 3 described above, and thus detailed description thereof will be omitted. By this process, the width (180 degrees) of the clipping range in the clipping process described later in S502 and S503 does not change, and the position of the clipping range in the VR image before clipping is changed. When it is determined in S404 that the display range changing function according to the posture detection is invalid (OFF), the process proceeds to S407 without proceeding to S405 and S406, and even if the posture detecting unit 213 detects the posture change, The display range is not changed according to the change of.

In S407 and S408, the CPU 201 changes the display magnification of the VR image 602 according to the pinch operation (magnification change instruction). This process is the same as S309 and S310 in FIG. 3 described above, and thus detailed description thereof will be omitted.

In S409, the CPU 201 determines whether or not a touch operation (whole hemisphere instruction; magnification change instruction) has been made on the hemisphere whole display button 609 based on the information notified from the touch panel 206a. If there is a touch operation for designating the hemisphere whole display button 609, the process proceeds to S410, and if not, the process proceeds to S413. In S410, the CPU 201 determines whether or not the current display magnification has already become the magnification of hemispherical whole display (hemispherical magnification). The determination may be made by referring to the current display magnification or the current display mode. If the hemispherical magnification has already been reached, the process proceeds to S412 to restore the original display magnification. If the hemispherical magnification is not set, the process proceeds to S411, and the display is switched to the hemispherical magnification. That is, every time the hemisphere whole display button 609 is touched, the hemisphere whole display (hemisphere magnification display) is switched on and off.

In step S411, the CPU 201 records the current display magnification before the display of the entire hemisphere in the memory 202, and then displays the entire hemisphere. The entire hemisphere display is a display at a magnification such that the entire cutout range when the pointer 603 is cut out around the position currently indicated is the display range on the display 205. In the present embodiment, since the range of 180 degrees is set as the cut-out range, the display magnification including the hemisphere range is set as the hemispherical magnification. However, when the cut-out range is not 180 degrees, the display range is not limited to the hemisphere range. The entire output range is displayed at a magnification that fits within the display range.

FIG. 7A shows a display example of the entire hemisphere display on the cutout edit screen. When an instruction to display the entire hemisphere is issued from the normal display magnification in the VR view as shown in FIG. 6B (the initial magnification initially displayed after starting the display in the VR view), FIG. Switch to the display magnification of (a magnification lower than the initial magnification) with one touch (without stopping at other magnifications). The VR image 602 in FIG. 7A is the same image as the VR image 602 displayed in FIG. 6B and is displayed at the magnification of the whole hemisphere display. Since a wide range of 180 degrees or more that is originally mapped on a sphere is included in the flat display range, a VR image 602 with a distorted periphery, such as that captured by a fisheye lens, is displayed. On the VR image 602, a shaded area 702 is displayed so as to be superimposed. The shaded area 702 indicates a range outside the cutout range (outside the range indicated by the pointer 603 and 180 degrees) of the displayed VR image 602, and a range to be deleted in the cutout image ( Excluded range) is shown. In the VR image 602, the user can confirm the part included in the cutout range by looking at the non-shaded part 701 at the center part, and can also check the excluded range by looking at the shaded part 702. Therefore, it is possible to specify the cutout range while confirming what range is included in the cutout range and where is not included in the cutout range. As a result, it is possible to easily perform an operation such as setting the cutout range so that a passing vehicle and a person fall within the cutout range while keeping the river, which is the main subject, within the cutout range.

Since it is sufficient to know that the shaded area 702 is the excluded range, the display form is not limited to the oblique line, and the excluded range may be indicated by other display forms such as a semi-transmissive mask, halftone dot mesh, and monochrome display. .. Further, even if the exclusion range is not shown, only the boundary between the cutout range and the exclusion range may be displayed by a circular line or the like. Furthermore, the exclusion range may not be displayed, and the entire extraction range may be displayed within the display range. Even during the display of the entire hemisphere, the processing according to various operations described in the processing of FIG. 4 can be performed. That is, it is possible to change the display range by changing the posture or touch-moving and adjust the desired range to be the cutout range. Further, it is possible to determine the cutout range by a process corresponding to a touch operation on the cutout button 604 described later in S425 to S431. The shaded area 702 indicating the exclusion range may be displayed in the case of the whole hemisphere display corresponding to the touch operation on the whole hemisphere display button 609, and may not be displayed when the display magnification is reduced from the normal magnification by pinch-in. Further, the shaded area 702 indicating the exclusion range may be displayed in the case of the whole hemisphere display corresponding to the touch operation on the hemisphere whole display button 609, and may be displayed even when the display magnification is reduced from the normal magnification by pinch-in. Good.

In step S412, the CPU 201 refers to the original display magnification before switching, which is held in the memory 202, from the state in which the entire hemisphere display is performed, and restores the display state in the original display magnification. That is, for example, the display state of FIG. 7A is switched to the display state of FIG. 6B. As a result, the peripheral portion is distorted and displayed small, and is switched to a display state in which there is no distortion or little distortion and the central portion included in the cutout range is displayed large.

In S413, the CPU 201 determines whether or not there is a touch operation on the two-screen confirmation button 610 based on the information notified from the touch panel 206a. If there is a touch operation for designating the two-screen confirmation button 610, the process proceeds to S414, and if not, the process proceeds to S415. In S414, the CPU 201 performs a two-screen display process. The two-screen display process will be described later with reference to FIGS. 8 and 9.

In S415, the CPU 201 determines, based on the information notified from the touch panel 206a, whether or not there is a touch operation on the edge fine adjustment button 611 (a fine adjustment instruction for starting the fine adjustment of the display range). If there is a touch operation for designating the edge fine adjustment button 611, the process proceeds to S416, and if not, the process proceeds to S417. In S416, the CPU 201 performs edge fine adjustment processing. The edge fine adjustment process will be described later with reference to FIGS. 10 and 11.

In S417, the CPU 201 determines whether or not there is a touch operation on the unnecessary portion designation button 612 based on the information notified from the touch panel 206a. If there is a touch operation for designating the unnecessary portion designation button 612, the process proceeds to S418, and if not, the process proceeds to S419. In S418, the CPU 201 performs unnecessary portion designation processing. The unnecessary portion designation processing will be described later with reference to FIGS.

In S419, the CPU 201 determines, based on the information notified from the touch panel 206a, whether or not there is a touch operation on the elevation/depression angle valid/invalid button 613 (a switching instruction for switching the elevation/depression angle valid/invalid). If there is a touch operation for designating the elevation/depression angle valid/invalid button 613, the process proceeds to S420, and if not, the process proceeds to S423. In S420, the CPU 201 refers to the setting information held in the memory 202, and determines whether or not the current setting before the touch operation on the elevation/depression angle valid/invalid button 613 is accepted is the elevation/depression angle. If it is determined that the elevation/depression angle is valid, the process proceeds to S421, and if it is invalid, the process proceeds to S422. In step S<b>421, the CPU 201 sets the elevation/depression angle valid/invalid setting from valid to invalid, and records the setting state in the memory 202. In step S422, the CPU 201 sets the elevation/depression angle valid/invalid setting by changing it from invalid to valid, and records the setting state in the memory 202. When the elevation/depression angle valid/invalid setting is changed, the display form of the elevation/depression angle valid/invalid button 613 is also updated to show the changed setting state. When the elevation/depression angle valid/invalid setting is disabled in S421, the directions in which the display range of the VR image 602 can be changed in S403 and S406 are limited to the azimuth direction, and the display range is changed in the elevation/depression direction. You may not be allowed. That is, even if the left and right scrolls (the display range is changed in the azimuth direction) are scrolled according to the touch move or the posture is changed, the vertical scroll (the display range is changed in the elevation and depression direction) is not restricted. good.

In S423, the CPU 201 determines whether or not there is a touch operation on the posture detection ON/OFF button 614 based on the information notified from the touch panel 206a. If there is a touch operation for designating the posture detection ON/OFF button 614, the process proceeds to S424, and if not, the process proceeds to S425. In S424, the CPU 201 changes the setting state of the display range changing function according to the posture detection. If it is valid (on) before the change, it is changed to invalid (off), and the setting information indicating the changed setting is recorded in the memory 202. If it is invalid (off) before the change, it is changed to valid (on), and the setting information indicating the changed setting is recorded in the memory 202. When the setting state of the display range changing function according to the posture detection is changed, the display form of the posture detection ON/OFF button 614 is also updated so as to show the changed setting state.

In S425, the CPU 201 determines, based on the information notified from the touch panel 206a, whether or not a touch operation (cutout instruction, reference position specifying operation) has been performed on the cutout button 604. If the touch operation for designating the cutout button 604 is performed (when the cutout instruction is received), the process proceeds to S426, and if not, the process proceeds to S432.

In S426, the CPU 201 determines whether or not the unnecessary portion is registered by the unnecessary portion specification process described later. If the unnecessary portion is registered, the process proceeds to S428, and if the unnecessary portion is not registered, the process proceeds to S427. In S427, the CPU 201 performs cutout preview processing. The cutout preview process will be described later with reference to FIG. Alternatively, without performing the process of S426, if it is determined in S425 that the touch operation on the cutout button 604 is performed, the process may proceed to S427. Further, if it is determined in S425 that the cutout button 604 has been touched without performing the processes of S426 and S427, the process proceeds to S535 of FIG. 5 to be described later to perform the process of saving the cutout VR image. Good. The processing of S428 to S431 is processing when unnecessary portions are registered, and is the same processing as S1215 to S1218 of FIG.

In S432, the CPU 201 determines whether or not an operation of ending the cutout edit process and returning to the process of the VR view display of FIG. 3 has been performed. If the return operation has not been performed, the process returns to S402 to repeat the process. If the return operation has been performed, the process of FIG. 4 is terminated and the original VR view display is performed, and the process proceeds to S313 of FIG.

FIG. 5 shows a flowchart of the cutout preview process described above in S427 of FIG. This processing is realized by loading the program recorded in the non-volatile memory 203 into the memory 202 and executing it by the CPU 201. The cutout preview process of FIG. 5 is a process for confirming whether or not the cutout range designated by the user in the cutout editing process of FIG. 4 can be confirmed.

In step S501, the CPU 201 refers to the setting information stored in the memory 202 and determines whether the current setting is the effective elevation/depression angle. It should be noted that what is referred to here is a setting item whose setting can be changed by a touch operation on the elevation/depression angle valid/invalid button 613. If it is determined that the elevation/depression angle is valid, the process proceeds to S502, and if it is invalid, the process proceeds to S503.

In S502, the CPU 201 is based on the azimuth and elevation angles of the current display range (the cutout edit screen of the process described in FIG. 4, or the display range immediately before the cutout button 604 is touched on the preview screen described later). The cutout range is determined and the cutout process is performed. More specifically, a 180-degree range (a range of 90 degrees in each direction from the center) centered on the center of the display range (the position where the pointer 603 is displayed) is cut out from the 360-degree VR image, A 180-degree VR image (edited VR content) is generated. This processing will be described later with reference to FIGS. 16(a) and 16(c).

In step S<b>503, the CPU 201 causes the azimuth and horizontal direction (elevation) of the current display range (the cutout edit screen of the process described in FIG. 4, or the display range immediately before the cutout button 604 is touched on the preview screen described later). The cutting range is determined based on the depression angle of 0 degree), and the cutting process is performed. At this time, the cutout range is determined based on the horizontal direction (direction perpendicular to the zenith axis) regardless of the elevation/depression angle of the display range before cutout. More specifically, a 180-degree range centered on one point determined by the azimuth angle corresponding to the center of the display range (the position where the pointer 603 is displayed) in the horizontal direction is cut out from the VR image of 360 degrees. , A VR image of 180 degrees (edited VR content) is generated. This processing will be described later with reference to FIGS. 16(a) and 16(b).

In step S<b>504, the CPU 201 displays a preview screen on the display 205 on which the cutout result in step S<b>502 or S<b>503 can be confirmed. 6(e), 6(f), and 6(g) show display examples of the preview screen.

6E, the cutout button 604 is touched in the state of FIG. 6B (the display range of the VR image 602 is the range of FIG. 6B), and the cutout preview process is performed. It is a display example of a preview screen in the case of transition. In the circular display direction guide at the lower right of the screen, in addition to the fan-shaped display range 605 and the non-display range 606, an exclusion range 622 indicating a range outside the cutout range (exclusion area) (black in FIG. 6E) is displayed. The painted area is displayed. This allows the user to identify that there is no image in the range corresponding to the exclusion range 622 in all the 360-degree directions of the VR image 602. Further, the display form of the pointer 603 is changed and the pointer 621 is displayed. On the preview screen, the pointer 621 is displayed as an indicator indicating the center of the specified cutout range. Therefore, the pointer 621 is not fixed to the center of the display 205 but is displayed at a position indicating the center of the cutout range (effective video range) of the VR image 602. Therefore, with the change of the display range, the display position of the pointer 621 also moves in conjunction with the movement of the display position at the center of the cutout range (effective video range) of the VR image 602. Other display items having the same reference numerals as those in FIG. 6B are the same display items as those in the cutout edit screen. The two-screen confirmation button 610 and the unnecessary portion designation button 612 are not displayed on the preview screen. A reproduction start direction designation button 623, a save button 624, and a cancel button 625 are displayed as display items that are not displayed on the cutout edit screen but are displayed on the preview screen. The reproduction start direction designation button 623 is an instruction icon for designating a portion that becomes a display range when the VR image after cutout is first displayed next time. The center of the display range at the time when the playback start direction designation button 623 is touched (the direction designation instruction for designating the playback start direction) is the display range when the cut-out VR image is first displayed next time. It is the center, that is, the above-mentioned display start direction in the VR image after being cut out. The save button 624 is an instruction icon for confirming the cutout range and giving an instruction to record the image file of the VR image after cutout. The cancel button 625 is an instruction icon for returning to the cutout edit screen without saving the VR image in the current cutout range.

In S505 and S506, the CPU 201 changes the display range according to the touch move with respect to the area in which the VR image 602 is displayed. The details of the processing are the same as S305 and S306 in FIG. 3 described above. As described above, when the display range changes, the display position of the pointer 621 on the display 205 also changes. Even if the display range of the VR image changes, the relative positional relationship of the pointer 621 with respect to the VR image does not change.

FIG. 6F shows a display example when the display range is changed by touch-moving to the left from the state of FIG. 6E. The display range of the VR image 602 is changed, and the angle information 608 and the ranges indicated by the display range 605 and the non-display range 606 of the display direction guide are changed accordingly. Of the display direction guide, the part of the exclusion range included in the display range is displayed in a pattern 626 in a distinguishable manner. The VR image 602 is scrolled so as to move to the left side as compared with the state shown in FIG. 6E, and the right side region of the VR image is newly set as the display range as compared with the state shown in FIG. 6E. .. That is, the display range has moved to the right. In the example of FIG. 6F, as can be seen from the angle information 608 and the display direction guide, the display range is changed in a direction pointing approximately 90 degrees to the right as compared with the display range of FIG. 6E. Since the clipping is performed in a range of 180 degrees centered on the position indicated by the pointer 621, the exclusion range 630 begins to appear in the display range. The exclusion range 630 has only to be identified as a range in which a valid image does not exist, and may be, for example, a single color display, a predetermined pattern, or a gradation display form. The vehicle that passes in the direction in which the license plate can be seen, which has been described as an example in which the user wants to exclude from the image after cutting in FIG. 6C, is excluded as an exclusion range 630 in FIG. It can be confirmed that the VR image is not included.

FIG. 6G shows a display example when the display range is changed by further touch-moving to the left from the state of FIG. 6F. The display range of the VR image 602 has changed, and the angle information 608 and the range indicated by the display direction guide are also changed accordingly. In the example of FIG. 6G, as can be seen from the angle information 608 and the display direction guide, the display range is changed to a direction substantially right behind (180 degrees to the right) as compared with the display range of FIG. 6E. ing. The VR image 602 is not included in the display range, and the exclusion range 630 occupies the entire display range. The photographer himself described as an example in which the user wants to exclude from the image after cutting in FIG. 6D is excluded as the exclusion range 630 in FIG. 6G and is included in the VR image after cutting. It can be confirmed that not.

The processing of S507 to S515 is the processing of changing the display range according to the posture change, the processing of changing the display magnification according to the pinch operation, and the display processing with the hemispherical magnification according to the hemisphere whole instruction, and the processing of S404 in FIG. 4 described above. Since the processing is the same as the processing from S412 to S412, the description thereof is omitted. In addition, in the hemisphere whole display in S514, unlike the hemisphere whole display in S411, the exclusion range (the portion indicated by the hatched portion 702 in FIG. 7A) is not displayed. FIG. 7B shows a display example of the whole hemisphere display on the preview screen. Since only the effective video range of the cut-out VR image excluding the exclusion range is displayed, the portion indicated by the hatched portion 702 in FIG. 7A is not displayed. That is, of the VR image before clipping, the portion outside the effective video range after clipping is not displayed. By this display, the user can confirm what kind of subject is located in the central portion of the VR image after cutout and what subject is included in the cutout range within one screen. When a touch operation is performed on the cutout button 604 in the display state of FIG. 7A, a transition is made to the display state of FIG. 7B. When the hemisphere whole display button 609 is touched in the display states of FIGS. 6(e) to 6(g), the display state of FIG. 7(b) transits.

The processes of S516 and S517 are processes corresponding to the touch operation on the edge fine adjustment button 611 and are the same as the processes of S415 and S416 described above, and thus the description thereof will be omitted.

S518 to S521 are processes corresponding to the touch operation on the elevation/depression angle valid/invalid button 613, and are the same processes as S419 to S422 described above, and therefore description thereof will be omitted. If the elevation/depression angle is switched from valid to invalid in step S520, the process proceeds to step S503 to change the cutout range. Note that when the process proceeds from S520 to S503, the cutout range is not set based on the display range at that time, but the cutout button 604 that is a factor that determines the cutout range at S520. It is performed based on the display range when the operation is performed. That is, the azimuth of the center of the cut-out range at the time of S520 remains unchanged, and the range of 180 degrees from the VR image of 360 degrees (global range) is set so that the elevation angle at the center of the cut-out area becomes the horizontal direction. Hemisphere range) is cut out. Further, when the elevation/depression angle is changed from invalid to valid in S521, the process proceeds to S502 and the cutting range is changed. In addition, when the process proceeds from S521 to S502, the cutout range is not set based on the display range at that time, but the cutout button 604 that is a factor for determining the cutout range at S521 is set. It is performed based on the display range when the operation is performed. That is, the azimuth at the center of the cutout range at the time of S521 remains the same, and the elevation angle at the center of the cutout range is the factor that determines the cutout range at the time of S521, when the cutout button 604 is operated. The VR image of 180 degrees is cut out from the VR image of 360 degrees so that the elevation angle is in the display range.

The processes of S522 and S523 are processes corresponding to the touch operation on the posture detection ON/OFF button 614, and are the same as the processes of S423 and S424 described above, and therefore description thereof will be omitted.

In S524, the CPU 201 determines whether or not a touch operation has been performed on the reproduction start direction designation button 623 based on the information notified from the touch panel 206a. If the playback start direction designation button 623 is touched, the process proceeds to S525, and if not, the process proceeds to S526. In S525, the CPU 201 determines the display start direction of the VR image after cutout as the center of the current display range, and records the position information indicating the center of the current display range in the VR image in the memory 202. This position information is recorded as information indicating the display start direction in the attribute information of the image file of the cut-out VR image in S535 described later.

In S526, the CPU 201 determines whether or not the cutout button 604 has been touched, based on the information notified from the touch panel 206a. If there is a touch operation for designating the cutout button 604, the process proceeds to S527, and if not, the process proceeds to S532.

In S527, the CPU 201 determines whether or not the unnecessary portion is registered by the unnecessary portion specification processing described later. If the unnecessary portion is registered, the process proceeds to S528, and if the unnecessary portion is not registered, the process proceeds to S501. Subsequently, the process proceeds from S501 to S502 or S503, and the cutout range is changed based on the display range at the time when it is determined in S526 that the cutout button 604 is touched. As described above, in the present embodiment, when it is desired to change the cutout range on the preview screen, the cutout button 604 is touched to display the display range at that time, more specifically, the display at that time. The cutout range can be modified based on the position of the center of the range. When the cutout is performed again in S502 or S503, the preview screen is updated in S504 to show the corrected cutout range. The cutout range may not be modified on the preview screen. In this case, the cut-out button 604 is not displayed on the preview screen or is displayed in a display form such as gray-out that indicates inoperability, and the determination in S526 is not performed. Even so, when the user wants to change the cutout range by looking at the preview screen, the user performs a touch operation on the cancel button 625, returns to the cutout edit screen, and then specifies the cutout range again. The cutting range can be changed by doing this. The processing of S528 to S531 is processing when an unnecessary portion is registered and is the same processing as S1215 to S1218 of FIG.

In S532, the CPU 201 determines whether or not there is a touch operation on the cancel button 625, based on the information notified from the touch panel 206a. If there is a touch operation on the cancel button 625, the process advances to step S533 to return to the cutout edit process (cutout edit screen) described above with reference to FIG. If the cancel button 625 is not touched, the process proceeds to S534.

In S534, the CPU 201 determines whether or not the touch operation on the save button 624 is performed based on the information notified from the touch panel 206a. If the save button 624 is touched, the process proceeds to S535, and if not, the process proceeds to S505 and repeats.

In step S535, the CPU 201 generates an image file that can be displayed in the VR view from the VR image that has been cut out in the cutout range confirmed on the preview screen, and records the image file in the recording medium 208. Here, it is assumed that the VR image of 360 degrees before being cut out is subjected to the cutting process in S502 or S503, and the image held in the memory 202 or the recording medium 208 is recorded as an image file. However, at the time of S535, the image processing unit 204 may be controlled from the VR image of 360 degrees before clipping and the VR image of 180 degrees may be again clipped and recorded. Also, the image file of the cut-out VR image is recorded as a new save file. However, it may be overwritten and saved in the image file of the VR image of 360 degrees before clipping. The image file of the cut-out VR image is recorded in a format conforming to the VR180 standard. When the image file of the cut-out VR image is recorded, the processing of FIG. 5 is ended, the process returns to S301 of FIG. 3, and the VR view display of the cut-out source VR image of 360 degrees is performed. It should be noted that after the processing of FIG. 5 is completed, the VR view display of the newly saved cut-out VR image of 180 degrees may be performed.

The cut-out VR image (image file that can be displayed in the VR view) recorded in S535 will be described. If the VR image that has been cut out is a still image, the image that is obtained by transforming the image within the cutout range into a rectangle by the equirectangular cylindrical projection and storing it is generated in multiple files in one file in the multi-picture format. And record. When a VR image (original image) of 360 degrees before being cut out is displayed as a normal plane image instead of a VR view, an image drawn by the equirectangular projection in a rectangle as shown in FIG. Become. When a range of 180 degrees is cut out from the original image by the above-described processing, when the image is displayed as a normal plane image instead of the VR view, it is drawn in an approximately equilateral cylindrical projection as shown in FIG. 18B. The recorded image is recorded. At this time, the same image obtained by copying the image within the cutout range is recorded in the same file as a right-eye image and a left-eye image, respectively, even if there is no parallax. Note that two 180-degree ranges are cut out from one 360-degree VR image before cut-out so as to have a pseudo parallax (that is, the range between the left-eye image and the right-eye image is slightly different). May be. In this case also, the cutout range is determined based on the display range when the cutout button 604 is touch-operated. The file name of the VR image of the clipped still image is, for example, “123456.vr.jpg”, and the three-character character string “.vr” is described before the extension “.jpg”. When the image thus saved as a still image file is reproduced and displayed in the VR view, the image in FIG. 18B is displayed in a hemisphere-mapped state. FIG. 18D shows an example in which a VR image file of a clipped still image is reproduced and displayed in the twin-lens VR view (display example on the display 205 in a state where the VR goggles are not attached).

If the cut-out VR image is a moving image, the images (moving images) in which the images in the cutting range are mapped in a circle or an ellipse instead of the equirectangular projection are arranged side by side (side-by-side) in the image of one moving image. Generate and record the created video file. At this time, the same image obtained by copying the image within the cutout range is recorded side by side as a right-eye image and a left-eye image, respectively, even if there is no parallax. When such a moving image is displayed as a normal plane image instead of the VR view, it is displayed as shown in FIG. Note that two 180-degree ranges are cut out from one 360-degree VR image before cut-out so as to have a pseudo parallax (that is, the range between the left-eye image and the right-eye image is slightly different). May be. In this case also, the cutout range is determined based on the display range when the cutout button 604 is touch-operated. The file name of the VR image of the cut-out moving image is, for example, “123456.vr.mp4”, and the character string “.vr” of three characters is described before the extension “.mp4”. FIG. 18D shows a frame display example (display example on the display 205 not attached to the VR goggles) when the VR image file of the cut-out moving image is reproduced and displayed in the twin-lens VR view. It is similar to the example shown.

FIG. 8 shows a flowchart of the two-screen display process of S414 of FIG. 4 described above. This processing is realized by loading the program recorded in the non-volatile memory 203 into the memory 202 and executing it by the CPU 201.

In step S<b>801, the CPU 201 initially displays a two-screen display on the display 205. FIG. 9A shows a display example of the initial display of the two-screen display on the display 205. The display screen of the two-screen display is divided into an upper region 901 that displays the cutout range 903 of the VR image 602 and a lower region 902 that shows the exclusion range 904 of the same VR image 602. In the initial display, a distorted circular image is displayed in the circular display area of the upper area 901 so that the entire cutout range is within the display range. This display range is the same display range as the entire hemisphere display described with reference to FIG. Further, in the initial display, a distorted circular image is displayed in the circular display area of the lower area 902 so that the entire exclusion range is within the display range. This display range corresponds to the range of the hemisphere that is discarded after cutting out. Other display items having the same reference numerals as those described above are the same as those described above. By looking at the screen in which the entire clipping range and the entire exclusion range are displayed at the same time, the user can see what subject is included in the clipping range included after clipping in the VR image of 360 degrees before clipping. , It is possible to easily confirm what kind of subject is included in the exclusion range. In the VR view of the normal cutout edit screen, if the display magnification is larger than the whole hemisphere display, only a part of the cutout range is displayed, and the display range is always the center of the cutout range. Range boundaries are not displayed. Therefore, the boundary cannot be confirmed. Further, when the display magnification is larger than the whole hemisphere display on the preview screen, only a part of the cutout range is displayed, so that only one part of the boundary between the cutout range and the excluded range can be confirmed. When the cutout range is adjusted by looking at only one part of the boundary, even if the desired part is contained inside the cutout range in the displayed boundary, it is reversed in the vicinity of the undisplayed boundary. It is possible that the subject you want goes out of the clipping range. It is necessary to change the display range multiple times or to make a large change in order to confirm whether or not the desired portion is fit around all the boundaries. On the other hand, in the two-screen display as shown in FIG. 9A, the user can confirm the entire cutout range and the entire excluded range without changing the display range. Further, the user can adjust the cutout range and the exclusion range by touch move while making such confirmation. Therefore, it is possible to prevent mistakes such as putting a subject that should be included in the cutout range in the exclusion range without noticing its existence, or putting a subject that should be excluded in the cutout range in the cutout range without noticing its existence. can do. Note that the exclusion range displayed in the lower region 902 is a semi-transparent mask of a predetermined color, color conversion such as display in monochrome or sepia, and shading so that it is easy to understand that it is a part that is discarded after being cut out. Etc. may be given.

In step S<b>802, the CPU 201 determines whether a touch move (video range changing operation) has been performed in the upper area 901 or the lower area 902. If it is determined that there is a touch move, the process proceeds to S803, and if not, the process proceeds to S804. In step S<b>803, the CPU 201 changes the part that is the cutout range and the part that is the exclusion range in the VR image 602. Specifically, the CPU 201 changes the display range of the VR image 602 displayed in one of the upper area 901 and the lower area 902 where the touch move is performed, following the movement of the touch position. At this time, if the elevation/depression angle is set to be valid, the display range is changed by following both the horizontal movement component and the vertical movement component of the touch move. If the elevation/depression angle is disabled, the display range is moved according to the horizontal movement component of the touch move, but the display range is not moved according to the vertical movement component, and the pointer 603 is always 0 degrees (horizontal direction). ), and the display range does not move up and down. Further, the display range of the VR image 602 displayed in the other area is changed in association with the change of the display range in the area of the VR image 602 where the touch move is performed, and the display area of the VR image 602 is changed in the area where the touch move is performed. The hemisphere part on the opposite side that is not in the display range. As a result, the relative position of the pointer 603 (the center position of the upper area) displayed at the fixed position (the predetermined position on the screen of the display 205) on the display 205 with respect to the VR image 602 changes. The CPU 201 sets the range of the hemisphere centered on the position corresponding to the pointer 603 in the VR image 602 as the cutout range.

In S804, the CPU 201 determines whether a pinch operation has been performed in the upper area 901. If it is determined that the pinch operation is performed in the upper area 901, the process proceeds to S805, and if not, the process proceeds to S807. In step S805, when the CPU 201 enlarges or reduces the image of the exclusion range in the lower area 902 in step S809 to be described later than the initial display in step S801, the display magnification of the image of the exclusion range and the entire exclusion range are displayed. To the initial display magnification. In step S806, the CPU 201 changes the display magnification of the image of the cutout range portion of the VR image 602 displayed in the upper area 901 in response to the pinch operation performed in the upper area 901, and enlarges the image. Or reduce and display. By enlarging and displaying the image of the cutout range, the user can check the details of the subject included in the cutout range.

In S807, the CPU 201 determines whether or not there is a pinch operation in the lower area 902. If it is determined that the pinch operation is performed in the lower area 902, the process proceeds to S808, and if not, the process proceeds to S810. In S808, when the CPU 201 enlarges or reduces the image in the cutout range in the upper area 901 from the initial display in S801 in S806, the CPU 201 sets the display magnification of the image in the cutout range to the entire cutout range. Is displayed, that is, the initial display magnification is reset. In step S<b>809, the CPU 201 changes the display magnification of the image of the exclusion range portion of the VR image 602 displayed in the lower region 902 in accordance with the pinch operation performed in the lower region 902, and enlarges the image. Reduce and display. By enlarging and displaying the image of the exclusion range, the user can confirm the details of the subject included in the exclusion range.

FIG. 9B shows a display example when the image displayed in the lower region 902 is enlarged in response to the pinch-out in the lower region 902. The range displayed in the lower area 902 is a part of the exclusion range instead of the entire range, but since it is enlarged from the initial display in FIG. 9A, the state of the subject can be easily confirmed. When a pinch operation is performed in the upper area 901 from this state, the display magnification is reset by the processing in S805, and the display magnification in the lower area 902 returns to the display magnification of the initial display shown in FIG. 9A. In this way, either the cutout range or the exclusion range is always displayed at a display magnification that fits the entire image, so that the boundary part between the cutout range and the exclusion range can always be confirmed in either of the distorted circular images. To do.

Note that, in S804 to S809, when the magnification is changed according to the pinch operation in the two-screen display process, only the display magnification is changed, and the cutout range is not changed. That is, the relative position of the pointer 603 with respect to the VR image 602 does not change.

In S810, the CPU 201 determines whether or not a touch operation has been performed on the cutout button 604. If there is a touch operation for designating the cutout button 604, the process advances to step S811, and the CPU 201 performs the cutout preview process of FIG. 5 described above. In this case, in the first S502 or S503 in the cutout preview processing, the CPU 201 causes the center of the display range displayed in the upper area 901 at the time when the cutout button 604 is touched (the position indicated by the pointer 603). Cut out so that is in the center of the cutting range. In addition, when the unnecessary portion is registered, the same processing as S426 to S431 in FIG. 4 described above may be performed.

In S812, the CPU 201 determines whether or not a touch operation has been performed on the save button 624. If there is a touch operation for designating the save button 624, the process proceeds to step S813, and if not, the process proceeds to step S814. In step S813, the CPU 201 performs clipping so that the center of the display range (the position indicated by the pointer 603) displayed in the upper area 901 when the save button 624 is touched is the center of the cropping range. More specifically, the CPU 201 cuts out a range of 180 degrees centering on the center of the display range in the upper area 901 (a range of 90 degrees in all directions from the center) from the VR image of 360 degrees. Then, the CPU 201 records the cut-out VR image on the recording medium 208 as an image file that can be displayed in the VR view. The recording process after the cutout is the same process as S535 described above. When the process of S813 ends, the two-screen display process ends and the process returns to S301 of FIG.

In S814, the CPU 201 determines whether or not a touch operation has been performed on the cancel button 625. When there is a touch operation for designating the cancel button 625, the two-screen display process is ended, and the process returns to S415 to display the cutout edit screen. At this time, the display range (relative positional relationship of the pointer 603 with respect to the VR image 602) adjusted in the two-screen display process is carried over to the cutout edit screen. If there is no touch operation on the cancel button 625, the process returns to S802 and is repeated.

Note that the two-screen display process is described as being displayed when the user performs a touch operation on the two-screen confirmation button 610 on the cutout edit screen, but according to the provisional setting instruction of the cutout range. Alternatively, a two-screen display may be performed as a preview display of the cutout range. For example, in response to the cutout button 604 being touched in S425 of FIG. 4, the two-screen display process of FIG. 8 may be performed instead of the cutout preview process of FIG.

FIG. 10 shows a flowchart of the edge fine adjustment processing of S416 of FIG. 4 and S517 of FIG. 5 described above. This processing is realized by loading the program recorded in the non-volatile memory 203 into the memory 202 and executing it by the CPU 201. The edge fine adjustment process of FIG. 10 is a process for enlarging and displaying the vicinity of the boundary between the cutout range and the exclusion range of the VR image and allowing the user to finely adjust the cutout range.

In S1001, the CPU 201 initially displays an edge fine adjustment screen for cutting out an image from the VR view.

FIG. 11A shows a display example of the edge fine adjustment screen displayed on the display 205 immediately after the transition to the edge fine adjustment processing. FIG. 11A shows an example of horizontal display of the edge fine adjustment screens of two types of horizontal display and vertical display. In the main area 1101, a central portion of the cutout range, which is a part of the same VR image 602 as that displayed on the cutout edit screen or the preview screen, is displayed. In the main area 1101, since the VR image 602 is displayed at a display magnification that does not distort the peripheral portion, the end of the cutout range (the boundary between the cutout range and the exclusion range) is not displayed. The pointer 603 is displayed at a fixed position in the main area 1101 (center of the main area 1101). The display position of the background VR image is determined so that the pointer 603 is at the same position as the position displayed on the cutout edit screen with respect to the VR image. By looking at the main area 1101, the user can confirm what is in the center of the cutout range. A left side area 1102 and a right side area 1103, which are end display areas, are displayed below the main area 1101 side by side. In the left side area 1102 and the right side area 1103, a part of the cutout range including the boundary between the cutout range and the excluded range is displayed.

In the left-side area 1102, a part of the cutout range is displayed as a boundary portion (a part of the range including the end of the cutout range). Immediately after the initial display and reset, the left boundary portion at the vertical position indicated by the pointer 603 (the position of the elevation angle indicated by the pointer 603) is displayed. In the left area 1102, an image area 1102a indicating the inside of the clipping range and an exclusion area 1102b indicating the outside of the clipping range (within the exclusion range) are divided into left and right parts and displayed adjacent to each other, and the boundary between the clipping range and the exclusion range is displayed. You can see what's going on. The positional relationship between the image area 1102a and the exclusion area 1102b in the left area 1102 is fixed, and does not change even if there is a touch move described later.

In the right area 1103, a part of the cutout range, which is 180 degrees opposite to the range displayed in the left area 1102, is a boundary part (a boundary part discontinuous with the range displayed in the left area 1102). Is displayed. Immediately after the initial display and reset, the right boundary portion at the vertical position indicated by the pointer 603 (the position of the elevation angle indicated by the pointer 603) is displayed. In the right area 1103, an image area 1103a indicating the inside of the clipping range and an exclusion area 1103b indicating the outside of the clipping range (within the exclusion range) are divided into left and right parts and displayed adjacent to each other, and the boundary between the clipping range and the exclusion range is displayed. You can see what's going on. The positional relationship between the image area 1103a and the exclusion area 1103b in the right area 1103 is fixed, and does not change even if there is a touch move described later. That is, even if the position of the cutout range in the VR image before cutout is changed according to the user operation, the position on the display 205 corresponding to the end of the cutout range is not changed. The position on the display 205 corresponding to the end of the cutout range is the boundary position between the image area 1103a and the exclusion area 1103b and the boundary position between the image area 1102a and the exclusion area 1102b.

When the user sees such a display and, for example, wants to exclude the roadway portion of the VR image 602 into the exclusion range and keep the sidewalk portion within the cutout range, the desired portion is included in the cutout range. It is possible to easily check whether or not there is any and finely adjust the cutout range. In the VR view of the normal cutout edit screen, if the display magnification is larger than the whole hemisphere display, only a part of the cutout range is displayed, and the display range is always the center of the cutout range. Range boundaries are not displayed. Therefore, the boundary cannot be confirmed. Further, in the preview screen, when the display magnification is larger than the whole hemisphere display, only a part of the cutout range is displayed, and therefore, only one boundary between the cutout range and the exclusion range can be confirmed. If you adjust the clipping range by looking at only one boundary, even if the desired boundary is contained within the clipping range at the displayed boundary, you want to capture it in the opposite direction near the other boundary. It may happen that the is out of the cutout range. It is necessary to change the display range a plurality of times or to make a large change in order to confirm whether or not the desired portion is close to both boundaries. On the other hand, on the edge fine adjustment screen as shown in FIG. 11A, the user confirms the boundary between the cutout range and the excluded range without changing the display range, and the user can find the desired range within the cutout range. You can check whether the part fits. Further, on the edge fine adjustment screen as shown in FIG. 11A, the user can simultaneously confirm a plurality of boundary portions (boundary portions at both ends), and whether or not the desired portion is within the cutout range. Confirmation can be done in a short time and with a small number of operations.

A horizontal/vertical switching button 1104, which is a touch icon, is displayed in the upper left portion of the left area 1102 (the upper portion of the exclusion area 1102b). By displaying the horizontal/vertical switching button 1104 at this position, the visibility of the image within the cutout range displayed in the image area 1102a is not hindered. Further, even when the horizontal display and the vertical display are switched, the horizontal/vertical switching button 1104 is displayed at a position that does not hinder the visibility of the image within the cutout range after switching without changing the display position. To do. The horizontal/vertical switching button 1104 may be displayed in the lower right portion of the right side area 1103 (lower portion of the exclusion area 1103b). Even at this position, it is possible to display the image in the cutout range without hindering the visibility of the image in both the horizontal display and the vertical display. A reset button 1105, which is a touch icon, is displayed in the lower right part of the right side region 1103 (lower part of the exclusion region 1103b). At this position, the reset button 1105 can be displayed in both the horizontal display and the vertical display without hindering the visibility of the image within the cutout range. Note that the reset button 1105 may be displayed in the upper left part of the left side region 1102 (the upper part of the exclusion region 1102b) for the same reason.

FIG. 11C shows a display example of the vertical direction display from the two types of edge fine adjustment screens of the horizontal direction display and the vertical direction display. The display content of the main area 1101 is the same as the horizontal display. A left side area 1112 and a right side area 1113, which are end display areas, are displayed below the main area 1101 side by side. In the left area 1112 and the right area 1113, a part of the cutout range including the boundary between the cutout range and the excluded range is displayed.

In the left area 1112, a part of the boundary portion of the cutout range is displayed. Immediately after initial display and reset, the upper boundary portion at the horizontal position indicated by the pointer 603 (the position of the azimuth angle indicated by the pointer 603) is displayed. In the left area 1112, an image area 1112a indicating the inside of the clipping range and an exclusion area 1112b indicating the outside of the clipping range (within the exclusion range) are vertically divided and displayed adjacent to each other, and the boundary between the clipping range and the exclusion range is displayed. You can see what's going on. The positional relationship between the image area 1112a and the exclusion area 1112b in the left area 1112 is fixed, and does not change even if there is a touch move described later.

In the right area 1113, a part of the cutout range, which is 180 degrees opposite to the range displayed in the left area 1112, is displayed. Immediately after the initial display and reset, the lower boundary portion at the horizontal position indicated by the pointer 603 (the position of the azimuth angle indicated by the pointer 603) is displayed. In the right area 1113, an image area 1103a indicating the inside of the clipping range and an exclusion area 1103b indicating the outside of the clipping range (within the exclusion range) are vertically divided and displayed adjacent to each other, and the boundary between the clipping range and the exclusion range is displayed. You can see what's going on. The positional relationship between the image area 1113a and the exclusion area 1113b in the right area 1113 is fixed, and does not change even if there is a touch move described later.

A horizontal/vertical switching button 1104, which is a touch icon, is displayed in the upper left portion of the left area 1112 (the upper portion of the exclusion area 1112b). This display position is the same as that shown in FIG. This display position is an area where the exclusion area 1102b in FIG. 11A and the exclusion area 1112b in FIG. 11C overlap. A reset button 1105 is displayed in the lower right part of the right side region 1113 (lower part of the exclusion region 1113b). This display position is the same as that shown in FIG. This display position is an area where the exclusion area 1103b in FIG. 11A and the exclusion area 1113b in FIG. 11C overlap.

In step S1002, the CPU 201 determines whether or not the reset button 1105 is touched, based on the information notified from the touch panel 206a. If there is a touch operation for designating the reset button 1105, the process advances to step S1001 to return the display position changed in step S1014 or S1020 described below to the original display position. By this reset, the left and right borders of the upper and lower central portions of the cutout range are displayed in the horizontal display, and the upper and lower borders of the left and right center of the cutout range are displayed in the vertical display. If there is no touch operation on the reset button 1105, the process proceeds to S1103.

In S1003, the CPU 201 determines whether or not a touch operation has been performed on the horizontal/vertical switching button 1104 based on the information notified from the touch panel 206a. If there is a touch operation for designating the horizontal/vertical switching button 1104, the process proceeds to S1004, and if not, the process proceeds to S1005.

In step S1004, the CPU 201 switches to vertical display when the edge fine adjustment screen (edge display area) is horizontal display, and switches to horizontal display when it is vertical display. For example, when the horizontal/vertical switching button 1104 is touched in the state shown in FIG. 11A, the display is switched to that shown in FIG. 11C. If a touch operation is performed on the horizontal/vertical switching button 1104 in the state shown in FIG. 11C, the display is switched to that in FIG. 11A.

In S1005, the CPU 201 determines whether or not the touch move is performed in the main area 1101 based on the information notified from the touch panel 206a. More specifically, it is determined whether the touch move is performed by touching down on the main area 1101. If the touch move is started from the main area 1101, the process proceeds to S1006, and if not, the process proceeds to S1008.

In S1006, the CPU 201 changes the display range of the VR image displayed in the main area 1101 according to the slide operation by the touch move. This process is similar to the process of S306 of FIG. 3 described above.

In step S<b>1007, the CPU 201 displays the VR image in the left area 1102 and the right area 1103 in the horizontal direction in response to the slide operation by the touch move (that is, the portion displayed in the image area 1102 a and the image area 1103 a ). To change. In the case of vertical display, the display range of the VR image in the left area 1112 and the right area 1113 (that is, the portion displayed in the image area 1112a and the image area 1113a) is changed. This is because the cutout range is changed according to the touch move in the main area 1101, and the positional relationship between the cutout range and the exclusion range in the VR image changes.

In step S1008, the CPU 201 determines whether or not there is a touchdown on any of the left side area 1102, the right side area 1103, the left side area 1112, and the right side area 1113 excluding the touch button display area, based on the information notified from the touch panel 206a. judge. If there is a touchdown on any of the left side area 1102, the right side area 1103, the left side area 1112, and the right side area 1113 excluding the display area of the touch button, the process proceeds to S1009, and if not, the process proceeds to S1023.

In step S1009, the CPU 201 clears (turns off) a move flag, which is held in the memory 202 and indicates a vertical touch move or a horizontal touch move, for initialization.

In S1010, the CPU 201 determines whether or not the touch move is performed in the vertical direction of the display 205 based on the information notified from the touch panel 206a. For example, each time there is a touch move of a unit movement amount, if the vertical movement component is larger than the horizontal movement component, it is determined that the vertical touch movement has been performed. If the touch move is performed in the vertical direction, the process proceeds to step S1011. If not, the process proceeds to step S1016.

In step S1011, the CPU 201 determines whether or not the left/right move flag, which is held in the memory 202 and indicates that the touch move in the left/right direction is in progress, is turned on. If it is determined that the left/right move flag is ON, the vertical movement component is ignored, and the process advances to step S1016. If it is determined that the left and right move flags are off, the process proceeds to S1012.

In step S1012, the CPU 201 sets the up/down move flag indicating that the touch move is being performed in the up/down direction to on (records in the memory 202 as on), and the process proceeds to step S1013.

In S1013, the CPU 201 determines whether the current display mode is the horizontal display of the horizontal display and the vertical display of the edge fine adjustment screen (edge display area). If it is the horizontal display, the process proceeds to S1014, and if it is the vertical display, the process proceeds to S1015.

In step S1014, the CPU 201 updates the display position of the VR image in the left area 1102 and the right area 1103 along the boundary of the cutout range. This means that when the hemisphere is designated as the cutout range out of the entire sphere to which the VR image of 360 degrees is mapped, the display range moves along the cutout boundary of the hemisphere which becomes a circle. Therefore, the positional relationship between the cutout range and the exclusion area in the VR image does not change, and the range of the VR image displayed in the image area 1102a and the range of the VR image displayed in the image area 1103a change. The image area 1102a and the image area 1103a maintain the relationship that even if the display range is changed, the portions of the cutout boundary that are different by 180 degrees are displayed. Therefore, when the user performs a touch move up and down by 180 degrees, it is possible to confirm the entire range of the boundary of the currently set cutout range. Since the cutout range is not changed in the process of S1014, the display content of the main area 1101 is not changed.

In FIG. 11B, a downward touch move is performed on the left side region 1102 from the state of FIG. 11A, and the image portion displayed in the image region 1102a is moved downward (moves the display range upward). A display example when scrolling is shown. The part displayed in the left side region 1102 moves along the boundary of the cutout range from the state of FIG. 11A in which the horizontal (horizontal direction) boundary part of the VR image is displayed, and In (b), a boundary portion in the sky direction is displayed in the left area 1102. The portion displayed in the right side area 1103 is an image of the ground direction on the side of the cutout range, which is 180 degrees opposite to the sky direction displayed in the left side area 1102.

In S1015, the CPU 201 changes the display range of the VR image displayed in the image area 1112a and the image area 1113a according to the upper and lower touch moves. This means that the position of the boundary of the cutout range with respect to the VR image before cutout moves according to the upper and lower touch moves. Since the angle of the cutout range is maintained at 180 degrees, the display range of both the image area 1112a and the image area 1113a changes regardless of which position of the left side area 1112 and the right side area 1113 is touched up and down. .. For example, when the display range of the image region 1112a changes due to the movement of the boundary in the direction of narrowing the cutout range, the boundary is moved in the direction of widening the cutout range such that the angle of the cutout range is maintained at 180 degrees. The display range of the image area 1113a also changes. As a result, the cutout range is changed, and the range displayed in the main area 1101 is also changed. The user performs a touch move up and down while checking the extent of the cutout range and the exclusion range near the boundary (image area 1112a and image area 1113a) to precisely adjust the cutout range. be able to. In this process, the size of the clipping range (180 degrees) does not change, and the position of the clipping range in the VR image before clipping is changed.

In step S1016, the CPU 201 determines, based on the information notified from the touch panel 206a, whether or not a touch move has been performed in the horizontal direction of the display 205. For example, every time there is a touch move of the unit movement amount, if the horizontal movement component is larger than the vertical movement component, it is determined that the touch movement has been performed in the horizontal direction. If the touch move is performed in the left-right direction, the process proceeds to S1017, and if not, the process proceeds to S1022.

In step S1017, the CPU 201 determines whether the up/down move flag, which is held in the memory 202 and indicates that the touch move in the up/down direction is being performed, is on. If it is determined that the up/down move flag is ON, the horizontal movement component is ignored, and the process advances to step S1022. If it is determined that the up/down move flag is off, the process proceeds to S1018.

In step S1018, the CPU 201 sets the left/right move flag indicating that the touch move in the left/right direction is in progress to ON (records it in the memory 202 as ON), and proceeds to step S1019.

In S1019, the CPU 201 determines whether the current display mode is the vertical direction display of the horizontal direction display and the vertical direction display of the edge fine adjustment screen (edge display area). If it is the vertical display, the process proceeds to S1020, and if it is the horizontal display, the process proceeds to S1021.

In S1020, the CPU 201 updates the display position of the VR image in the left area 1112 and the right area 1113 along the boundary of the cutout range. This is the same processing as S1014 described above. However, the display direction of the VR image in the left area 1112 and the right area 1113 is different from that in S1014 by 90 degrees. Since the cutout range is not changed in the process of S1020, the display content of the main area 1101 is not changed.

In FIG. 11D, a left-side touch move is performed on the right side area 1113 from the state of FIG. 11C, and the image portion displayed in the image area 1113a is moved to the left (the display range is rightward in the direction of the image area 1113a). The following is an example of the display when scrolling to move to. The portion displayed in the right side region 1113 moves along the boundary of the cutout range from the state of FIG. 11C in which the vertical lower side (ground direction) boundary portion of the VR image is displayed, In FIG. 11D, the boundary portion in the horizontal direction (horizon direction) is displayed in the right area 1113. The part displayed in the left area 1112 is an image in the horizontal direction (horizon direction) opposite to the direction displayed in the right area 1113 by 180 degrees in the boundary of the cutout range.

In S1021, the CPU 201 changes the display range of the VR image displayed in the image area 1102a and the image area 1103a according to the left and right touch moves. This means that the position of the boundary of the cutout range with respect to the VR image before cutout moves in accordance with the left and right touch moves. Since the angle of the cutout range is maintained at 180 degrees, the display range of both the image area 1102a and the image area 1103a changes regardless of which position of the left area 1102 and the right area 1103 the touch move is performed. .. For example, when the display range of the image region 1102a changes due to the movement of the boundary in the direction of narrowing the cutout range, the movement of the boundary in the direction of widening the cutout range maintains the angle of the cutout range at 180 degrees. The display range of the image area 1103a also changes. As a result, the cutout range is changed, and the range displayed in the main area 1101 is also changed. The user performs a touch move to the left and right while confirming how much is within the cutout range and what is the excluded range near the boundary (image area 1102a and image area 1103a), and the cutout range is adjusted strictly. be able to. In this process, the size of the clipping range (180 degrees) does not change, and the position of the clipping range in the VR image before clipping is changed.

In S1022, the CPU 201 determines whether touch-up has been performed based on the information notified from the touch panel 206a. If touch-up has been performed, the process proceeds to S1023, and if not, the process returns to S1010 to repeat the process.

In S1023, the CPU 201 determines whether or not there is a touch operation on the return button 1106 based on the information notified from the touch panel 206a. If there is no touch operation for designating the return button 1106, the process returns to S1002 and repeats the processing, and if there is a touch operation for the return button 1106, the edge fine adjustment processing ends.

When the edge fine adjustment processing is finished, the cut-out range adjusted by the edge fine adjustment processing is inherited even in the processing after the edge fine adjustment processing is finished. Therefore, for example, when the trimming range is finely adjusted by the edge fine adjustment process and the edge fine adjustment process is finished and the process returns to the trimming edit process (the trimming edit screen) in FIG. 4, the display range is not changed thereafter. By touching the cutout button 604 on, the cutout can be performed within the cutout range adjusted by the fine edge adjustment processing. Alternatively, the cutout instruction may be received on the edge fine adjustment screen, and when the cutout instruction is received on the edge fine adjustment screen, the process may directly proceed to S426 or S427. Further, the cutout saving instruction may be received on the edge fine adjustment screen, and when the cutout saving instruction is received on the edge fine adjustment screen, the process may directly proceed to S535.

FIG. 12 shows a flowchart of the unnecessary portion designation processing of S418 of FIG. 4 described above. This processing is realized by loading the program recorded in the non-volatile memory 203 into the memory 202 and executing it by the CPU 201.

In step S1201, the CPU 201 displays the unnecessary portion designation screen on the display 205. FIG. 13A shows a display example of the unnecessary portion designation screen. Display items having the same reference numerals as those described above are the same as those described above (display items on the cutout edit screen and the preview screen).

A pointer 1301 is fixedly displayed at the center of the unnecessary portion designation screen and is an indicator for designating an unnecessary portion (position of unnecessary subject, position of subject to be excluded) in the VR image 602. In order to prevent confusion with the pointer 603 and the pointer 621, the pointer 603 and the pointer 621 are displayed in a different display form (color, shape, pattern, etc.). However, the pointer 1301 functions as a pointer that indicates the center of the cutout range when a touch operation is performed on the save button 624 displayed on the unnecessary portion designation screen described later. The unnecessary part registration button 1302 is a touch button for designating and registering an unnecessary part. When the touch operation for designating the unnecessary portion registration button 1302 is performed, the position information of the position indicated by the pointer 1301 at that point in the VR image 602 is registered as an unnecessary portion (recorded in the memory 202). The unnecessary portion information deletion button 1303 is a touch button for deleting information on the unnecessary portion that has already been registered. When a touch operation for designating the unnecessary portion information deletion button 1303 is performed, one piece of registered unnecessary portion information is deleted (cleared), and registration as an unnecessary portion is canceled. The margin setting button 1304 is a touch button for setting a margin of an invalid selection range starting from an unnecessary portion. The end button 1305 is a touch button that receives an instruction to end the unnecessary portion designation processing and return to the cutout editing processing.

Steps S1202 and S1203 are display range changing processing according to the touch move, and are the same processing as steps S305 and S306 in FIG. Even if the display range of the VR image 602 is changed by the touch move, the display range of the pointer 1301 is not changed. Further, when the unnecessary portion is registered, if the selected invalid range is included in the changed display range, the selection invalid range is displayed (highlighted) in the displayed VR image. This will be described later with reference to FIG.

Steps S1204 and S1205 are display magnification changing processing in accordance with the pinch operation, and are the same processing as S309 and S310 in FIG. 3, so detailed description thereof will be omitted. In addition, when the unnecessary portion is registered, if the selection invalid range is included in the display range after the display magnification is changed, the selection invalid range of the displayed VR image is displayed so that the selection invalid range can be identified.

In S1206, the CPU 201 determines whether or not a touch operation has been performed on the unnecessary portion registration button 1302. If the unnecessary portion registration button 1302 is touched, the process proceeds to S1207, and if not, the process proceeds to S1208. In step S<b>1207, the CPU 201 sets the position information (the elevation angle information and the azimuth angle information) of the position indicated by the pointer 1301 in the VR image 602 as an unnecessary part and registers corresponding unnecessary part information. As unnecessary portion information, position information and a registration number (+1 from every registration from 1) are recorded in the memory 202. If the position indicated by the pointer 1301 has already been registered as an unnecessary part, the process of S1207 is skipped.

When the unnecessary portion is newly registered, the CPU 201 sets, as the selection invalid range, an area of an angle obtained by combining the cutout angle and the margin angle around the position information of the newly registered unnecessary portion. If the reference position of the cutout range is set in the selection invalid range, the unnecessary part that has already been registered will be included in the cutout range. It can also be said. Even if the reference position of the cutout range is set in a range other than the selection invalid range (selection effective range), the unnecessary points that have already been registered are not included in the cutout range, so the range other than the selection invalid range It can be said that it is an appropriate range suitable for setting the reference position of the cutout range. The cutout angle is 180 degrees in the present embodiment. The margin angle can be arbitrarily changed by the user in S1210 and S1211, which will be described later, but the initial value is 10 degrees. When the cutting angle is 180 degrees and the margin angle is 10 degrees, a range of 100 degrees around the newly registered unnecessary portion (90 degrees of the cutting range 180 + 10 degrees), a total of 200 degrees, It becomes the selection invalid range. If there is another unnecessary part that has already been registered and the selection invalid range has already been set, the selection invalid range that has been set is merged with the selection invalid range that has already been set. The selected range is set as the selection invalid range. If any position within the selection invalid range is set as the center of the cutout range, the subject (video) at the position designated as the unnecessary portion may be included in the cutout range. That is, each position within the selection invalid range is a position that should not be set as the center of the cutout range. Then, the CPU 201 performs identification display on the VR image 602 displayed on the display 205 so that the set selection invalid range can be identified.

FIG. 13B shows a display example of the identification display of the invalid selection range. In the displayed VR image 602, the selection invalid range 1307 is displayed in a identifiable manner with diagonal lines. The display form of the identification display is not limited to this, and may be a display with a semi-transmissive mask, a display by changing color such as monochrome or sepia. Further, although an example in which the selection invalid range is emphasized and displayed is described in the present embodiment, it is sufficient if the selection invalid range and the range other than the selection invalid range can be distinguished, and the range other than the selection invalid range is highlighted and displayed. Is also good. Further, the video in the selection invalid range of the VR image 602 may be hidden and the video outside the selection invalid range may be displayed.

In addition, in the unnecessary part registration processing of S1207, if the entire range of the VR image before clipping becomes the selection invalid range due to the newly registered unnecessary part, the new unnecessary part is not registered, An error message indicating that additional registration cannot be performed is displayed. An error indicating that additional registration cannot be performed without registering a new unnecessary part even when the width of the range (selection valid range) other than the selection invalid range in the VR image before clipping is less than the threshold value Display.

The flow until the selection invalid range is set will be described with reference to the image diagrams of FIGS. 14(a) to 14(g). A range 1450 in FIG. 14A indicates a region (display range) displayed on the display 205 in the VR image, and a pointer direction 1451 indicates a central direction of the display range viewed from the center of the virtual sphere. , Corresponding to the position indicated by the pointer 1301. The unnecessary portion candidate 1452 is a portion that the user considers to be an unnecessary portion (excluding portion) in the VR image for clipping. When there is an unnecessary portion for clipping, the user sets the pointer direction 1451 corresponding to the center of the display area of the display 205 to the unnecessary spot 1452 for clipping (a subject that is considered unnecessary for clipping). Then, the unnecessary portion registration button 1302 is touched.

FIG. 14B is an image diagram immediately after touching to register one unnecessary portion. The shaded selection invalid range 1453 represents a region corresponding to the cutout angle 1454 (180°) from the pointer direction 1451, and indicates the selection invalid range when the margin is 0 degree. As shown in FIG. 14C, when the position 1455 in the selection invalid range is designated as the center of the cutout, the cutout range becomes the range of the white part 1456 in FIG. 14D, and the unnecessary portion 1452 is included. I will end up. In addition, in FIG. 14D, the black-painted portion is an exclusion range.

When the margin angle is set, the selection invalid range is set to a range wider than 180 degrees (hemisphere) centering on the unnecessary portion and becomes like the selection invalid range 1457 in FIG. Further, when a plurality of unnecessary portions are registered as shown in FIG. 14F (an example in which two unnecessary portions 1452 and 1458 are registered), the selection invalid range is calculated centering on each of them and all selection invalidation is performed. The combined area is the selection invalid range.

Of the selection invalid range thus set (determined), the area displayed on the display 205 is displayed in a distinguishable manner as a selection invalid range 1307 in FIG. 13B. The identification display such as the invalid selection range 1307 is displayed not only on the unnecessary portion designation screen but also on the cutout edit screen of the cutout edit processing described in FIG. The user may be able to switch on/off (display/non-display) the identification display of the invalid selection range. Further, a boundary line between the selection invalid range and the range other than that may be displayed. Further, in the cutout edit screen, when the pointer 603 is in the selection invalid range, the position of the pointer 603 is set as the central designated portion of the cutout range by changing the display form of the pointer 603, displaying a warning icon, a warning message, or the like. The user may be notified of the unsuitability.

In S1208, the CPU 201 determines whether or not there is a touch operation on the unnecessary portion information deletion button 1303 based on the information notified from the touch panel 206a. If there is a touch operation for designating the unnecessary portion information deletion button 1303, the process proceeds to S1209, and if not, the process proceeds to S1210. In step S1209, the CPU 201 acquires unnecessary portion information from the memory 202. If no unnecessary part information is registered, the process is interrupted and the process proceeds to S1210. If it is registered, the display range of the VR image 602 is moved so that the angle of the position information with the largest registration number (the unnecessary part registered last) becomes the center of the display, and the unnecessary part information is displayed. Delete from memory 202. Then, the selection invalid range based on the deleted unnecessary portion is cleared (released). That is, the selection invalid range based only on the deleted unnecessary portion is removed from the merged selection invalid range, and the identification display of the removed portion is hidden (displayed as the selection valid range). When the unnecessary portion information is deleted when only one unnecessary portion information is registered, the shaded portion such as the selection invalid range 1307 shown in FIG. 13B disappears.

In S1210, the CPU 201 determines whether or not a touch operation has been performed on the margin setting button 1304, based on the information notified from the touch panel 206a. If there is a touch operation for designating the margin setting button 1304, the process proceeds to step S1211, and if not, the process proceeds to step S1211. In step S1211, the CPU 201 reads margin setting information from the memory 202, displays a margin setting dialog, and accepts a margin setting operation. FIG. 13C shows a display example of the margin setting dialog 1306. The background area other than the margin setting dialog 1306 may be displayed dark. The CPU 201 changes the set numerical value of the margin in response to a touch operation on the up and down arrow marks of the margin setting dialog 1306, a touch move operation up and down, and a numeric input operation performed by displaying a ten-key pad. The changed numerical value is stored in the memory 202. When the set numerical value of the margin is changed, the width of the selection invalid range based on the registered unnecessary portion information is also changed accordingly. The margin is set by setting the extent of the selection invalid range set based on one unnecessary portion to be wider than the cutout range (180 degrees in the present embodiment). The range of the numerical value of the margin setting information is 0 to (360-cutout angle), and when it exceeds the range, the numerical value is not changed. When a portion other than the margin setting dialog 1306 is touched, the margin setting dialog 1306 is closed (hidden).

In S1212, the CPU 201 determines whether or not there is a touch operation on the save button 624 based on the information notified from the touch panel 206a. If there is a touch operation for designating the save button 624, the process proceeds to step S1213, and if not, the process proceeds to step S1219.

In step S1213, the CPU 201 determines whether or not the unnecessary portion is registered in the memory 202. When one or more unnecessary parts are registered, the process proceeds to S1215, and when no one is registered, the process proceeds to S1214.

In step S<b>1214, the CPU 201 cuts out based on the azimuth angle and the elevation/depression angle at the center of the current display range, that is, the center of the cutout range (the position indicated by the pointer 1301 in the unnecessary portion designation screen). The range is determined and the cutting process is performed. More specifically, a 180-degree range centered on the center of the display range (a range of 90 degrees in all directions from the center) is cut out from the 360-degree VR image. Then, the cut-out VR image is recorded in the recording medium 208 as an image file that can be displayed in the VR view. The recording process after the cutout is the same process as S535 described above. When the process of S1214 is finished, the unnecessary portion designation process is finished and the process returns to S301 of FIG.

In step S1215, the CPU 201 determines whether the center of the current display range, that is, the center of the cutout range (the position indicated by the pointer 1301 in the unnecessary portion designation screen) is within the selection invalid range. Determine whether. If it is within the selection invalid range, the process proceeds to S1216, and if not, the process proceeds to S1214.

In step S<b>1216, the CPU 201 displays a warning on the display 205 notifying that the registered unnecessary portion will fit in the cutout range based on the display range when the save button 624 is touched. FIG. 13D shows a display example of the warning displayed in S1216. In the dialog 1308, an OK button 1309 and a cancel button 1310 are displayed together with a message “Do you want to enter unnecessary parts?”. By looking at this display, the user can recognize that the registered unnecessary portion is included in the cutout range based on the display range at the time when the save button 624 is touched.

In S1217, the CPU 201 determines whether or not a touch operation has been performed on the OK button 1309. If the OK button 1309 is touched, the process advances to step S1214. That is, since the user is aware that the unnecessary portion is included in the cutout range, the user cuts out in the cutout range including the unnecessary portion and saves the cut-out VR image. If there is no touch operation on the OK button 1309, the process proceeds to S1218.

In S1218, the CPU 201 determines whether or not there is a touch operation on the cancel button 1310. If there is no touch operation on the cancel button 1310, the process returns to S1217. If the cancel button 1310 is touched, the dialog 1308 is hidden and the process returns to step S1202 without performing cutout saving in the cutout range based on the display range at the time when the save button 624 is touched. repeat.

By the processes of S1215 to S1218, it is possible to prevent the user from inadvertently designating the center of the cutout range within the selection invalid range. If the pointer is within the selection invalid range without displaying the warning (that is, without performing the processing of S1216 to S1218), the cutout saving instruction may be ignored (invalid). Alternatively, only the warning may be displayed, and the cutout saving may not be performed without the selection of OK or cancellation.

In addition to the various display items described with reference to FIG. 13A, a jump button may be displayed. Then, when the pointer 1301 is within the selection invalid range, the VR image 602 is displayed so that the pointer 1301 is located outside the selection invalid range according to a touch operation (touch operation without movement of the touch position) on the jump button. The range may be changed.

Also, instead of or in addition to the unnecessary part registration button 1302 described in FIG. 13A, a necessary part registration button may be displayed. In response to a touch operation on the necessary portion registration button, the position of the VR image 602 indicated by the pointer 1301 at that time is recorded (registered) in the memory 202 as position information of the necessary portion. Then, an area other than the area of the angle obtained by subtracting the margin angle from the cutout angle is calculated as the selection invalid range with the position information of the necessary portion as the center. In the image diagram, when the necessary portion 1459 is specified in FIG. 14G, an area other than the cutout angle (example 180°)-margin angle (example 10°) area is invalidated like the selection invalid range 1460. Set as a range. When there are a plurality of unnecessary points or necessary points, the area including the selection invalid range calculated based on each of all designated positions (registered positions) is set as the selection invalid range.

In S1219, the CPU 201 determines whether or not there is a touch operation on the end button 1305. When there is no touch operation for designating the end button 1305, the process is returned to S1202 and repeated. When there is a touch operation for designating the end button 1305, the unnecessary portion designation process is terminated and the process proceeds to S419 in FIG.

15(a) to 15(d) and 16(a) to 16(c), the cutout process described in S502 when the elevation/depression angle is valid, and the elevation/depression angle invalid in S503 are described. The cutout process in this case will be described.

FIG. 15A shows a display example of the cutout edit screen on the display 205 (that is, a display example before specifying the cutout range). The reference direction (display start direction) is set to an azimuth angle of 0 degrees (for example, north) and an elevation/depression angle of 0 degrees (directions perpendicular to the zenith axis and the gravity direction axis, horizontal direction). FIG. 15A is a display example of the VR image 602 when the azimuth angle is 0 degree and the elevation/depression angle is about 60 degrees on the cutout edit screen. That is, the pointer 603 indicating the center of the display range is displayed in a direction that is upward (above the sky) with respect to the horizontal direction. Note that FIG. 15A is a display example when the above-described elevation/depression angle valid/invalid setting is invalid, and the elevation/depression angle valid/invalid button 613 is a display form indicating that the elevation/depression angle is invalid. Has become.

FIG. 16A is a diagram schematically showing the display range of FIG. 15A on a virtual sphere onto which a VR image of 360 degrees is mapped. It is possible to specify the position on the sphere with the azimuth of 360 degrees around the zenith axis 1602 and the elevation angle of 360 degrees around the horizontal axis perpendicular to the zenith axis 1602 centering on the viewing position 1601 of the viewer. The VR image of 360 degrees before clipping is mapped on the entire sphere, and the entire sphere becomes the effective video range 1603. FIG. 16A shows an example in which the line-of-sight direction 1605 is directed to a subject 1604 in the sky above the north (that is, an azimuth angle of 0 degree) as viewed from the viewer. That is, the display range 1607 is set at a position where the center of the display range 1607 intersects with the line-of-sight direction 1605. Note that the display range of FIGS. 16A to 16C is deformed and illustrated, and is narrower than the display range of FIGS. 15A to 15D. However, the center of the display range of FIGS. 16A to 16C is illustrated on the same basis as the center of the display range shown in FIGS. 15A to 15D. Therefore, the center of the display range 1607 is the position indicated by the pointer 603 in FIG. 15A, which is the cutout designated position 1606.

When the elevation angle is invalid and the display range is set to the range shown in FIGS. 15A and 16A and the cutout button 604 performs a cutout range designation operation, the cutout process is performed in S503. Is performed, and the display range shown in FIG. FIG. 15B is a display example of the preview screen immediately after the cutout button 604 is touched and the cutout range is set from the state of FIG. 15A. The elevation/depression angle of the display range is different from 60° before the clipping and is in the horizontal direction (elevation/depression angle 0°), and the pointer 621 is also displayed at the position of the elevation/depression angle 0°. The azimuth angle of the display range in FIG. 15B is 0 degree (north), which is the same as that in FIG. 15A before cutting. Since the azimuth angle is the same as that before cutting, if the azimuth angle of 90 degrees (east) is set as the display range before cutting, the display range of the preview screen immediately after and the center of the cutting range will also have the azimuth angle of 90 degrees ( East).

FIG. 16B is a diagram schematically showing the cutout range in the state of FIG. 15B on a virtual sphere onto which a VR image of 360 degrees is mapped. The white hemisphere on the right side of the sphere is a cutout range 1611 (effective image range) that is a VR image of 180 degrees after cutout. The hemisphere on the opposite side is an exclusion range 1612 (invalid video range) that is excluded from the VR image after being cut out and discarded. The cut-out range 1611 is a range of a hemisphere centered at the central position 1613. The central position 1613 is a position with an azimuth of 0 degrees indicated by the cutout specified position 1606 and an elevation angle of 0 degrees (horizontal) regardless of the cutout specified position 1606. The center position 1613 of the cutout range 1611 and the cutout specified position 1606 based on the line-of-sight direction 1605 and the display range 1607 immediately before the operation of specifying the cutout range do not match as illustrated, and the elevation angle is different. However, this is because the elevation/depression angle is set to be invalid. The cutting surface 1610 of the cutting range 1611 is parallel to the zenith axis 1602.

When the elevation angle is valid and the display range is at the positions shown in FIGS. 15C and 16A, if the cutout range is specified by the cutout button 604, the cutout process is performed in S502. Is performed, and the display range shown in FIG. Note that FIG. 15C has the same display range as FIG. 15A. Only the setting of the elevation/depression angle valid/invalid differs from FIG. 15(a), and in FIG. 15(c), the display form of the elevation/depression angle valid/invalid button 613 indicates that the elevation/depression angle is set. Has become. FIG. 15D is a display example of the preview screen immediately after the cutout button 604 is touch-operated and the cutout range is set from the state of FIG. 15C. The elevation/depression angle of the display range is the same as 60 degrees before the clipping, and the pointer 621 is also displayed at the position of the elevation/depression angle 60 degrees. The azimuth angle of the display range in FIG. 15D is 0 degree (north), which is the same as that in FIG. 15C before cutting.

FIG. 16C is a diagram schematically showing the display range and the cutout range in the state of FIG. 15D on a virtual sphere onto which a VR image of 360 degrees is mapped. A white hemisphere on the upper right side of the sphere is a cutout range 1611 (effective image range) that becomes a VR image of 180 degrees after cutout. The hemisphere on the opposite side is an exclusion range 1612 (invalid video range) that is excluded from the VR image after being cut out and discarded. The center position 1613 of the cutout range 1611 coincides with the cutout designated position 1606 in both azimuth and elevation. This is because the elevation/depression angle is set to be valid.

In cropping a VR image in this embodiment, the initial setting of valid/invalid elevation/depression is invalid, and cropping is performed within the cropping range as shown in FIG. 16B. In this cutout range, the effective image range is from directly above (90 degrees elevation) to directly below (90 degrees depression), so the viewer viewing on the HMD simply changes the face orientation (simply shakes his/her head). ), you can easily see the whole VR image after clipping. If, as shown in FIG. 16C, the effective image range extends beyond just above and to the rear side, the direction of the face should be above just above in order to view the back side beyond just above. You need to turn to the back. In order to do this, not only the person's body needs to be shaken, but also the body must be bent, the upper body must be twisted, and the body must be turned to a different position and turned to the back. That is, it is physically difficult to view the VR image after cutout by changing the display range according to the posture change. On the other hand, regardless of the elevation/depression angle of the display range when designating the cutout range, cutting out the range centered on the horizontal direction as shown in FIG. 16B reduces the physical burden. However, it is possible to generate a VR image in which it is easy to see the whole. Also, since the human eye is in a position where the front in the horizontal direction can be seen when facing the front, setting the elevation angle at the center of the visual field (that is, the cut-out range) to the horizontal direction causes less discomfort due to the structure of the human body. ..

Further, in cutting out a VR image in the present embodiment, if the setting of valid/ineffective of the elevation/depression angle is set to be valid, the cutting is performed in the cutting range as shown in FIG. 16C. This cut-out range is suitable depending on the content of the subject, such that the whole starry sky can be overlooked in the VR image after cut-out and unnecessary ground portions can be excluded, or a wide range of the room viewed from the ceiling can be seen.

Whether the clipping is performed within the clipping range 1611 shown in FIG. 16(b) or the clipping range 1611 shown in FIG. 16(c) is performed by changing the elevation/depression angle valid/invalid setting. It is hard for users to imagine and hard to understand. Therefore, it is advisable to display a guidance using a schematic diagram so that the user can easily understand what kind of difference the cutout range will have depending on the setting of the elevation/depression angle valid/invalid.

FIG. 17A shows a display example of guidance display of the cutout range on the display 205. The guidance dialog 1700 of FIG. 17A is predetermined when, for example, the elevation/depression angle valid/invalid button 613 is touched and the elevation/depression angle setting is switched from invalid to valid (S422 in FIG. 4, S521 in FIG. 5). The time (for example, 5 seconds) is displayed. In the dialog 1700, a graphic 1701 showing a schematic diagram before clipping, a graphic 1702 showing a schematic diagram after clipping, and a message 1704 are displayed. A graphic 1701 is a guide using a schematic diagram of a sphere, and shows a schematic diagram when a position above the elevation/depression angle of 0 degrees is designated as a clipping designation position (position indicated by an arrow) on the sphere before clipping. A graphic 1702 is a guide using a schematic diagram of a sphere, and shows a cutting range when the elevation/depression angle setting is valid. After cutting, a hemisphere based on the elevation/depression angle of the designated cutting position is cut out. Is shown. A message 1704 indicates that the setting of the elevation/depression angle of the cutout range is valid. Instead of the graphic 1701, a graphic 1703 of FIG. 17B showing a cutout range when the elevation/depression angle setting is valid may be displayed.

FIG. 17B shows a display example of guidance display of the cutout range on the display 205. The guidance dialog 1700 of FIG. 17B is predetermined when, for example, the elevation/depression angle valid/invalid button 613 is touched and the elevation/depression angle setting is switched from valid to invalid (S421 in FIG. 4, S520 in FIG. 5). The time (for example, 5 seconds) is displayed. In the dialog 1700, a graphic 1701 showing a schematic diagram before clipping, a graphic 1703 showing a schematic diagram after clipping, and a message 1705 are displayed. A graphic 1701 is a guide using a schematic diagram of a sphere, and shows a schematic diagram in the case where a position above the elevation/depression angle of 0 degree is designated as a clipping designation position (position indicated by an arrow) before clipping, This is the same as the graphic 1701 of 17(a). A graphic 1703 is a guide using a schematic diagram of a sphere, and shows a cutout range when the elevation/depression angle setting is invalid. After cutting, the sphere is vertically divided regardless of the elevation/depression angle at the designated cutout position. It shows that the hemisphere (the hemisphere whose center is horizontal) is cut out. A message 1705 indicates that the setting of the elevation/depression angle of the cutout range is invalidated. Instead of the graphic 1701, a graphic 1702 of FIG. 17A may be displayed which shows the cutout range when the elevation/depression angle setting is invalid.

The guides as shown in FIGS. 17A and 17B may be displayed not only when the setting of the elevation/depression angle is switched but also when an explanation regarding the setting of the elevation/depression angle is displayed in an electronic manual or the like. .. Further, as the information indicating the setting state, for example, the elevation/depression angle valid/invalid button 613, a guide as shown in FIGS. 17A and 17B may be displayed. Instead of displaying the graphic 1701 indicating before cutout, the graphic indicating after cutout (the graphic 1702 when the elevation/depression angle is invalid and the graphic 1703 when the elevation/depression angle is valid) may be displayed.

Note that, in the above-described embodiment, an example in which a VR image in a narrower range is generated from a VR image captured by the image capturing apparatus has been described, but the original VR image is not limited to the VR image captured by the image capturing apparatus. The present invention is also applicable to the case where a narrower VR content is cut out and generated from VR content created by computer graphics (CG) or the like.

Further, the processes of the various flowcharts described above do not necessarily have to be performed in combination, and may be performed separately, and it is possible to employ a part of the above-described flowcharts in combination with other embodiments. is there. For example, the above-mentioned whole hemisphere display, two-screen display processing, edge fine adjustment processing, and the like can be performed without being combined with the cutout editing processing of FIG. 4 that specifies the cutout range by the method of specifying the center of the cutout range. This can be carried out by itself as a process for designating the cutout range. That is, for example, if it is determined in S311 of FIG. 3 that the touch operation has been performed on the cutout edit button, the hemisphere whole display described in FIGS. 7A and 7B is directly moved to set the cutout range. You may be able to specify. The cutout range may be designated by directly shifting to the two-screen display processing described in FIGS. 8 and 9 or the edge fine adjustment processing described in FIGS.

According to the present invention described above, a VR content in a narrower range can be generated from the VR content with good operability based on the user's operation of designating the cutout range.

Note that the various controls described as being performed by the CPU 201 may be performed by one piece of hardware, or a plurality of pieces of hardware (for example, a plurality of processors or circuits) may share the processing to control the entire apparatus. You may go.

Although the present invention has been described in detail based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various embodiments within the scope not departing from the gist of the present invention are also included in the present invention. included. Furthermore, each of the above-described embodiments is merely an example of the present invention, and the embodiments may be combined as appropriate.

Further, although cases have been described with the above embodiment as examples where the present invention is applied to the electronic device 200 (such as a smartphone), the present invention is not limited to this example. INDUSTRIAL APPLICABILITY The present invention can be applied to any electronic device that can be controlled to display an operation screen for cutting out a VR content in a narrower range from the VR content. For example, the present invention is applicable to personal computers, PDAs, mobile phone terminals, portable image viewers, printer devices, digital photo frames, music players, game machines, electronic book readers, video players and the like. Further, the present invention can be applied to a digital camera, a television device, a projection device, a tablet terminal, an AI speaker, a home electric device, an in-vehicle device, a medical device, and the like.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Claims (20)

An electronic device,
On the first screen, control is performed so that a part of the VR content having the first video range is displayed as a display range on the display unit, and the attitude of the electronic device is changed or the display range is changed by the user. Display control means for changing the position of the display range,
A receiving means for receiving a cutout instruction from the user,
Generating means for generating an edited VR content having a second video range which is a video range narrower than the first video range of the VR content;
A second range, which is wider than the partial range of the VR content and includes the entire second video range, is displayed on one screen so that the range of the second video range can be identified. An electronic device comprising: a control unit that controls to display a screen. On the second screen, the control means displays a range wider than the second video range including the entire second video range of the VR content before receiving the cutout instruction. The electronic device according to claim 1, wherein the electronic device is controlled so as to perform an identification display that enables identification of how far the second video range is. The electronic device according to claim 2, wherein the identification display is at least one of a display indicating a boundary of the second video range and a display capable of discriminating a range outside the second video range. machine. The control unit controls the second screen so that the outside of the second video range of the VR content is hidden after the cutout instruction is received. The electronic device according to item 3. The control unit controls to change the position of the second video range in the VR content in response to a user operation while displaying the second screen. 5. The electronic device according to any one of items 4 to 4. 6. The second screen is a screen displaying the VR content at a predetermined display magnification lower than the display magnification on the first screen. 6. Electronics. The control means is
When the VR image is displayed at the first display magnification on the first screen, the display magnification of the VR content is gradually reduced in three or more stages in accordance with a first user operation. Control and
Control to switch to the second screen in response to a second user operation different from the first user operation when the VR image is displayed at the first display magnification on the first screen The electronic device according to claim 6, wherein the electronic device comprises: The control means switches to the first screen in response to the second user operation while displaying the second screen, and a display magnification before switching to the second screen. The electronic device according to claim 7, wherein the electronic device is controlled to display the VR content by. 9. The electronic device according to claim 7, wherein the first display magnification is a display magnification when the display of the first screen is started. The electronic device according to claim 1, further comprising a recording unit configured to generate and record a file in which the edited VR content generated by the generation unit is stored. The electronic device according to claim 10, wherein the recording unit generates one still image file in which a plurality of images based on the second video range are stored. The electronic device according to claim 11, wherein a character string ".vr" is described before the extension in the file name of the still image file. 13. The electronic device according to claim 10, wherein the recording unit generates one moving image file in which a plurality of moving images based on the second image range are arranged. The electronic device according to claim 13, wherein a character string ".vr" is described before the extension in the file name of the moving image file. An item indicating the center position of the second video range at a predetermined position on the screen of the second screen regardless of the position of the second video range of the VR content on the second screen. 15. The electronic device according to claim 1, wherein the electronic device is controlled to display. A method of controlling an electronic device,
On the first screen, control is performed so that a part of the VR content having the first video range is displayed as a display range on the display unit, and the attitude of the electronic device is changed or the display range is changed by the user. A display control step of changing the position of the display range,
A receiving step for receiving a cutout instruction from the user,
A generation step of generating edited VR content having a second video range which is a video range narrower than the first video range of the VR content;
A second range, which is wider than the partial range of the VR content and includes the entire second video range, is displayed on one screen so that the range of the second video range can be identified. And a control step of controlling to display a screen. An electronic device,
On the first screen, control is performed so that a part of the VR content having the first video range is displayed as a display range on the display unit, and the attitude of the electronic device is changed or the display range is changed by the user. Display control means for changing the position of the display range,
Generating means for generating edited VR content having a second video range narrower than the first video range of the VR content, and the second video range of the VR content generated by the generating means. The second video range in the VR content is displayed according to a user operation in a state in which a part of the range including the end is displayed so that the end can be identified, and the part including the end is displayed. And a control means for controlling so as to change the position of the electronic device. A method of controlling an electronic device,
On the first screen, control is performed so that a part of the VR content having the first video range is displayed as a display range on the display unit, and the attitude of the electronic device is changed or the display range is changed by the user. A display control step of changing the position of the display range,
A generation step of generating an edited VR content having a second video range narrower than the first video range of the VR content; and a second video range of the VR content generated in the generation step. The second video range in the VR content is displayed according to a user operation in a state in which a part of the range including the end is displayed so that the end can be identified, and the part including the end is displayed. And a control step for controlling so as to change the position of the electronic device. A program for causing a computer to function as each unit of the electronic device according to any one of claims 1 to 15. A computer-readable storage medium that stores a program for causing a computer to function as each unit of the electronic device according to claim 1.