JP2021174317A - Electronic apparatus and control method therefor - Google Patents

Abstract

To enable a physical load applied to a user to be reduced without deteriorating the visibility of the user in the case of changing display areas according to the posture of the user.SOLUTION: An electronic apparatus includes: detection means for detecting a posture of a display part; display means for displaying a range being a portion of an image and including a position separated according to difference of the posture and reference direction of the display part from a reference position set in the image in the display part; and update means for updating the reference position to a position corresponding to a constant direction in the case of maintaining a state where the posture of the display part is directed to the constant direction different from the reference direction for a first prescribed time.SELECTED DRAWING: Figure 3

Description

The present invention relates to an electronic device and a control method for the electronic device.

In recent years, imaging devices capable of capturing images including a wider range than the viewing angle of humans, such as omnidirectional images and spherical images, have become widespread. For example, Patent Document 1 discloses that the size of the angle of view to be displayed on the display unit can be set among the angles of view wider than the viewing angle of a human being, and that an image at a position corresponding to the posture information of the user is displayed. There is.

Japanese Unexamined Patent Publication No. 2015-125502

When the image of the position corresponding to the posture information is displayed, the user has to keep the posture changed or change the size of the angle of view in order to see the area different from the front. By changing the size of the angle of view, the user can see the desired area while facing the front, but the area that the user wants to see is displayed smaller than the area displayed at the angle of view before the change. .. Further, when the angle of view is not changed, the image is viewed in the state where the posture is changed, which may impose a physical load on the user.

In view of the above problems, it is an object of the present invention to reduce the physical load on the user without deteriorating the visibility of the user when the display area is changed according to the posture of the user.

In order to solve the above problems, the electronic device of the present invention
A detection means for detecting the posture of the display unit and
A display means for displaying a range of a part of the image including a position separated from the reference position set in the image according to the difference between the posture of the display unit and the reference direction on the display unit.
It has an updating means for updating the reference position to a position corresponding to the certain direction when the state in which the posture of the display unit is directed to a certain direction different from the reference direction for a first predetermined time is maintained. It is characterized by that.

According to the present invention, when the display area is changed according to the posture of the user, the physical load on the user can be reduced without deteriorating the visibility of the user.

It is an external view and a block diagram of a digital camera. It is an external view and a block diagram of an electronic device. It is a figure explaining the reference position update process according to the posture of a display control device. It is a flowchart which illustrates VR view display processing. It is a flowchart which illustrates the reference position update process. It is a flowchart which illustrates the reference position update process which concerns on the modification.

<Embodiment>
(Digital camera configuration)
Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. FIG. 1A is a front perspective view (external view) of a digital camera 100 (imaging device) which is an electronic device. FIG. 1B is a rear perspective view (external view) of the digital camera 100. The digital camera 100 is an omnidirectional camera (omnidirectional camera).

The barrier 102a is a protective window for the front camera unit with the front of the digital camera 100 as the shooting range. The front camera unit is, for example, a wide-angle camera unit that covers a wide range of 180 degrees or more in the vertical and horizontal directions on the front side of the digital camera 100. The barrier 102b is a protective window for the rear camera unit whose shooting range is the rear of the digital camera 100. The rear camera unit is, for example, a wide-angle camera unit that covers a wide range of 180 degrees or more in the vertical and horizontal directions on the rear side of the digital camera 100.

The display unit 28 displays various information. The shutter button 61 is an operation unit (operation member) for giving a shooting instruction. The mode changeover switch 60 is an operation unit for switching various modes. The connection I / F 25 is a connector for connecting the connection cable to the digital camera 100, and an external device such as a smartphone, a personal computer, or a television is connected to the digital camera 100 by using the connection cable. The operation unit 70 is various switches, buttons, dials, touch sensors, etc. that receive various operations from the user. The power switch 72 is a push button for switching the power on / off.

The light emitting unit 21 is a light emitting member such as a light emitting diode (LED), and notifies the user of various states of the digital camera 100 by a light emitting pattern and a light emitting color. The fixing portion 40 is, for example, a tripod screw hole, and is used for fixing and installing the digital camera 100 with a fixing device such as a tripod.

FIG. 1C is a block diagram showing a configuration example of the digital camera 100. The barrier 102a covers the imaging system (photographing lens 103a, shutter 101a, imaging unit 22a, etc.) of the front camera unit to prevent the imaging system from becoming dirty or damaged. The photographing lens 103a is a lens group including a zoom lens and a focus lens, and is a wide-angle lens. The shutter 101a is a shutter having an aperture function for adjusting the amount of subject light incident on the imaging unit 22a. The image pickup unit 22a is an image pickup device (imaging sensor) composed of a CCD, a CMOS element, or the like that converts an optical image into an electric signal. The A / D converter 23a converts the analog signal output from the imaging unit 22a into a digital signal. The outer surface of the photographing lens 103a may be exposed without providing the barrier 102a, and the photographing lens 103a may prevent other imaging systems (shutter 101a and imaging unit 22a) from becoming dirty or damaged.

The barrier 102b covers the imaging system (photographing lens 103b, shutter 101b, imaging unit 22b, etc.) of the rear camera unit to prevent the imaging system from becoming dirty or damaged. The photographing lens 103b is a lens group including a zoom lens and a focus lens, and is a wide-angle lens. The shutter 101b is a shutter having an aperture function for adjusting the amount of subject light incident on the imaging unit 22b. The image pickup unit 22b is an image pickup device composed of a CCD, a CMOS element, or the like that converts an optical image into an electric signal. The A / D converter 23b converts the analog signal output from the imaging unit 22b into a digital signal. The outer surface of the photographing lens 103b may be exposed without providing the barrier 102b, and the photographing lens 103b may prevent other imaging systems (shutter 101b and imaging unit 22b) from becoming dirty or damaged.

A VR (Virtual Reality) image is imaged by the imaging unit 22a and the imaging unit 22b. The VR image is an image that can be displayed in VR (displayed in the display mode "VR view"). The VR image includes an omnidirectional image (omnidirectional image) captured by an omnidirectional camera (omnidirectional camera) and a panorama having a wider image range (effective image range) than the display range that can be displayed on the display unit 28 at one time. Images etc. shall be included. The VR image includes not only a still image but also a moving image and a live view image (an image acquired from a camera in almost real time). The VR image has a maximum image range of 360 degrees in the vertical direction (vertical angle, angle from the zenith, elevation angle, depression angle, altitude angle, pitch angle) and 360 degrees in the horizontal direction (horizontal angle, azimuth angle, yaw angle). Has (effective video range).

Further, even if the VR image is less than 360 degrees up and down and less than 360 degrees left and right, it is displayed at once on the display unit 28 or a wide angle of view (field of view) wider than the angle of view that can be taken by a normal camera. Images with an image range (effective image range) wider than the display range that can be displayed shall also be included. For example, an image taken with an omnidirectional camera capable of shooting a subject with a field of view (angle of view) of 360 degrees in the left-right direction (horizontal angle, azimuth angle) and a vertical angle of 210 degrees centered on the zenith (zenith) It is a kind of VR image. Further, for example, an image taken by a camera capable of shooting a subject with a viewing angle (angle of view) of 180 degrees in the horizontal direction (horizontal angle, azimuth angle) and a vertical angle of 180 degrees centered on the horizontal direction is a VR image. It is a kind. That is, an image having a field of view of 160 degrees (± 80 degrees) or more in each of the vertical and horizontal directions and a wider image range than a human can see at one time is a VR image. It is a kind of.

When this VR image is displayed in VR (displayed in the display mode "VR view"), the posture of the display device (display device that displays the VR image) is changed in the left-right rotation direction, so that the left-right direction (horizontal rotation direction) can be changed. You can watch seamless omnidirectional video. In the vertical direction (vertical rotation direction), a seamless omnidirectional image can be viewed within a range of ± 105 degrees from directly above (zenith), but there is no image in a range exceeding 105 degrees from directly above. It becomes a blank area. A VR image can also be said to be an "image in which the image range is at least a part of a virtual space (VR space)".

The VR display (VR view) is a display method (display mode) in which the display range can be changed to display an image of a viewing range according to the posture of the display device among the VR images. When a head-mounted display (HMD), which is a display device, is attached for viewing, an image in a visual field range corresponding to the orientation of the user's face is displayed. For example, in a VR image, a viewing angle (angle of view) centered on 0 degrees in the left-right direction (specific orientation, for example, north) and 90 degrees in the up-down direction (90 degrees from the zenith, that is, horizontal) at a certain point in time. Is displayed. From this state, if the posture of the display device is turned upside down (for example, when the display surface is changed from south to north), 180 degrees in the left-right direction (opposite direction, for example, south) in the same VR image, up and down. The display range is changed to an image with a viewing angle centered on 90 degrees (horizontal) in the direction. In the case where the user is watching the HMD, if the user turns his face from north to south (that is, turns to the back), the image displayed on the HMD also changes from the north image to the south image. be. By such a VR display, it is possible to visually provide the user with the feeling of being in the place in the VR image (in the VR space). A smartphone attached to VR goggles (head mount adapter) can be said to be a type of HMD.

The VR image display method is not limited to the above. The display range may be moved (scrolled) according to the user's operation on the touch panel, the direction button, or the like instead of the posture change. In VR display (when the display mode is "VR view"), in addition to changing the display range by changing the posture, the display range can be changed according to touch move to the touch panel, drag operation to the mouse, etc., pressing the direction button, etc. You may do so.

The image processing unit 24 performs resizing processing such as predetermined pixel interpolation and reduction and color conversion processing on the data from the A / D converter 23a and the A / D converter 23b or the data from the memory control unit 15. In addition, the image processing unit 24 performs predetermined arithmetic processing using the captured image data. The system control unit 50 performs exposure control and distance measurement control based on the calculation result obtained by the image processing unit 24. As a result, TTL (through-the-lens) AF (autofocus) processing, AE (autoexposure) processing, EF (flash pre-flash) processing, and the like are performed. The image processing unit 24 further performs a predetermined calculation process using the captured image data, and performs a TTL method AWB (auto white balance) process based on the obtained calculation result.

The image processing unit 24 performs basic image processing on two images (two fisheye images; two wide-angle images) obtained from the A / D converter 23a and the A / D converter 23b, and basically performs basic image processing. A single VR image is generated by performing joint image processing that combines two images that have undergone image processing. Further, the image processing unit 24 performs image cutting processing, enlargement processing, distortion correction, etc. for displaying the VR image in VR at the time of VR display or playback in the live view, and draws the processing result in the VRAM of the memory 32. Perform rendering.

In the connected image processing, the image processing unit 24 uses one of the two images as a reference image and the other as a comparison image, calculates the amount of deviation between the reference image and the comparison image for each area by pattern matching processing, and calculates the amount of deviation between the reference image and the comparison image for each area. The connection position connecting the two images is detected based on the amount of deviation. The image processing unit 24 corrects the distortion of each image by geometric transformation in consideration of the detected connection position and the lens characteristics of each optical system, and converts each image into an all-sky format (all-sky image format). Convert to an image. Then, the image processing unit 24 generates one spherical image (VR image) by synthesizing (blending) two images in the spherical format. The generated spherical image is, for example, an image using equirectangular projection, and the position of each pixel of the spherical image can be associated with the coordinates of the surface of the sphere (VR space).

The output data from the A / D converters 23a and 23b is written to the memory 32 via the image processing unit 24 and the memory control unit 15 or via the memory control unit 15 without the image processing unit 24. The memory 32 stores image data obtained by the imaging units 22a and 22b and converted into digital data by the A / D converters 23a and 23b, and image data to be output from the connection I / F 25 to an external display. The memory 32 has a storage capacity sufficient to store a predetermined number of still images, moving images for a predetermined time, and audio.

Further, the memory 32 also serves as a memory (video memory) for displaying an image. The image display data stored in the memory 32 can be output from the connection I / F 25 to an external display. A VR image captured by the imaging units 22a and 22b and generated by the image processing unit 24, which functions as an electronic viewfinder by sequentially transferring and displaying the VR image stored in the memory 32 to an external display. Can be realized, and live view display (LV display) can be performed. Hereinafter, the image displayed in the live view display is referred to as a live view image (LV image). In addition, the VR image stored in the memory 32 can be transferred to an external device (smartphone or the like) wirelessly connected via the communication unit 54 and displayed on the external device side to perform live view display (remote LV display). ..

The non-volatile memory 56 is a memory as a recording medium that can be electrically erased and recorded, such as EEPROM. The non-volatile memory 56 records constants, programs, and the like for the operation of the system control unit 50. The program referred to here is a computer program for executing various processes.

The system control unit 50 is a control unit having at least one processor or circuit, and controls the entire digital camera 100. The system control unit 50 realizes each process by executing the program recorded in the non-volatile memory 56 described above. The system memory 52 is, for example, a RAM, and in the system memory 52, constants and variables for operation of the system control unit 50, a program read from the non-volatile memory 56, and the like are expanded. The system control unit 50 also controls the display by controlling the memory 32, the image processing unit 24, the memory control unit 15, and the like. The system timer 53 is a time measuring unit that measures the time used for various controls and the time of the built-in clock.

The mode changeover switch 60, the shutter button 61, the operation unit 70, and the power switch 72 are used to input various operation instructions to the system control unit 50.

The mode changeover switch 60 switches the operation mode of the system control unit 50 to any one of a still image recording mode, a moving image shooting mode, a playback mode, a communication connection mode, and the like. The modes included in the still image recording mode include an auto shooting mode, an auto scene discrimination mode, a manual mode, an aperture priority mode (Av mode), a shutter speed priority mode (Tv mode), and a program AE mode. In addition, there are various scene modes, custom modes, etc. that are shooting settings for each shooting scene. From the mode changeover switch 60, the user can directly switch to any of these modes. Alternatively, after switching to the shooting mode list screen with the mode changeover switch 60, the mode may be selectively switched to any of the plurality of modes displayed on the display unit 28 by using another operation member. Similarly, the moving image shooting mode may include a plurality of modes.

The shutter button 61 includes a first shutter switch 62 and a second shutter switch 64. The first shutter switch 62 is turned on by a so-called half-press (shooting preparation instruction) during the operation of the shutter button 61, and the first shutter switch signal SW1 is generated. The system control unit 50 starts shooting preparation operations such as AF (autofocus) processing, AE (autoexposure) processing, AWB (auto white balance) processing, and EF (flash pre-flash) processing by the first shutter switch signal SW1. do.

The second shutter switch 64 is turned on when the operation of the shutter button 61 is completed, so-called full pressing (shooting instruction), and the second shutter switch signal SW2 is generated. The system control unit 50 starts a series of shooting processes from reading signals from the imaging units 22a and 22b to writing image data to the recording medium 90 by the second shutter switch signal SW2.

The shutter button 61 is not limited to an operating member that can be operated in two steps of full pressing and half pressing, and may be an operating member that can be pressed in only one step. In that case, the shooting preparation operation and the shooting process are continuously performed by pressing the button in one step. This is the same operation as when the shutter button that can be half-pressed and fully pressed is fully pressed (when the first shutter switch signal SW1 and the second shutter switch signal SW2 are generated almost at the same time).

The operation unit 70 is assigned various functions for each scene by selecting and operating various function icons and options displayed on the display unit 28, and acts as various function buttons. The function buttons include, for example, an end button, a back button, an image feed button, a jump button, a narrowing down button, an attribute change button, and the like. For example, when the menu button is pressed, various settable menu screens are displayed on the display unit 28. The user can intuitively make various settings by operating the operation unit 70 while looking at the menu screen displayed on the display unit 28.

The power switch 72 is a push button for switching the power on / off. The power supply control unit 80 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching a block to be energized, and the like, and detects whether or not a battery is installed, the type of battery, the remaining battery level, and the like. Further, the power supply control unit 80 is DC-based on the detection result and the instruction of the system control unit 50.
The DC converter is controlled to supply a required voltage to each unit including the recording medium 90 for a required period. The power supply unit 30 includes a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, an AC adapter, or the like.

The recording medium I / F18 is an interface with a recording medium 90 such as a memory card or a hard disk. The recording medium 90 is a recording medium such as a memory card for recording a captured image, and is composed of a semiconductor memory, an optical disk, a magnetic disk, or the like. The recording medium 90 may be an interchangeable recording medium that can be attached to and detached from the digital camera 100, or may be a recording medium built in the digital camera 100.

The communication unit 54 transmits and receives video signals, audio signals, and the like to and from an external device connected by a wireless or wired cable. The communication unit 54 can also be connected to a wireless LAN (Local Area Network) or the Internet. The communication unit 54 can transmit images (including LV images) captured by the imaging units 22a and 22b and images recorded on the recording medium 90, and can receive images and various other information from an external device. ..

The posture detection unit 55 detects the posture of the digital camera 100 with respect to the direction of gravity. Based on the posture detected by the posture detection unit 55, the image taken by the image pickup units 22a and 22b is an image taken by holding the digital camera 100 horizontally or an image taken by holding the digital camera 100 vertically. It is possible to determine whether or not there is. Further, it is possible to determine how much the images taken by the imaging units 22a and 22b are taken by tilting the digital camera 100 in the three axial directions (rotational directions) of the yaw direction, the pitch direction, and the roll direction. be. The system control unit 50 adds orientation information according to the posture detected by the posture detection unit 55 to the image file of the VR image captured by the image pickup units 22a and 22b, and rotates the image (tilt correction (zenith correction)). It is possible to adjust the orientation of the image so that it can be recorded. One sensor such as an acceleration sensor, a gyro sensor, a geomagnetic sensor, an orientation sensor, an altitude sensor, or a combination of a plurality of sensors can be used as the attitude detection unit 55. It is also possible to detect the movement (pan, tilt, lift, whether or not it is stationary, etc.) of the digital camera 100 by using an acceleration sensor, a gyro sensor, a direction sensor, or the like that constitute the attitude detection unit 55.

The microphone 20 is a microphone that collects the sound around the digital camera 100 recorded as the sound of a VR image (VR moving image) which is a moving image. The connection I / F25 is a connection plug to which an HDMI (registered trademark) cable, a USB cable, or the like for connecting to an external device and transmitting / receiving video is connected.

(Configuration of display control device)
FIG. 2A is an example of an external view of a display control device 200, which is a type of electronic device. The display control device 200 is a display device such as a smartphone. The display 205 is a display unit that displays images and various information. The display 205 is integrally configured with the touch panel 206a, and can detect a touch operation on the display surface of the display 205. The display control device 200 can display a VR image (VR content) in VR on the display 205. The operation unit 206b is a power button that accepts an operation of switching the power of the display control device 200 on and off. The operation unit 206c and the operation unit 206d are volume buttons for increasing or decreasing the volume of audio output from the speaker 212b, earphones connected to the audio output terminal 212a, an external speaker, or the like. The operation unit 206e is a home button for displaying the home screen on the display 205. The audio output terminal 212a is an earphone jack, and is a terminal that outputs an audio signal to an earphone, an external speaker, or the like. The speaker 212b is a built-in speaker that outputs sound.

FIG. 2B is a block diagram showing a configuration example of the display control device 200. The CPU 201, memory 202, non-volatile memory 203, image processing unit 204, display 205, operation unit 206, recording medium I / F 207, external I / F 209, and communication I / F 210 are connected to the internal bus 250. .. Further, the audio output unit 212 and the posture detection unit 213 are also connected to the internal bus 250. Each part connected to the internal bus 250 can exchange data with each other via the internal bus 250.

The CPU 201 is a control unit that controls the entire display control device 200, and includes at least one processor or circuit. The memory 202 includes, for example, a RAM (such as a volatile memory using a semiconductor element). The CPU 201 controls each part of the display control device 200 by using the memory 202 as a work memory according to a program stored in the non-volatile memory 203, for example. The non-volatile memory 203 stores image data, audio data, other data, various programs for operating the CPU 201, and the like. The non-volatile memory 203 is composed of, for example, a flash memory or a ROM.

The image processing unit 204 controls the image stored in the non-volatile memory 203 or the recording medium 208 based on the control of the CPU 201, the video signal acquired via the external I / F 209, and the image acquired via the communication I / F 210. Various image processing is applied to such as. The image processing performed by the image processing unit 204 includes A / D conversion processing, D / A conversion processing, image data coding processing, compression processing, decoding processing, enlargement / reduction processing (resizing), noise reduction processing, and color conversion. Processing etc. are included. It also performs various image processing such as panoramic development, mapping processing, and conversion of a VR image which is an omnidirectional image or a wide range image having a wide range of images even if it is not omnidirectional. The image processing unit 204 may be configured by a dedicated circuit block for performing specific image processing. Further, depending on the type of image processing, the CPU 201 may perform image processing according to a program without using the image processing unit 204.

The display 205 displays an image, a GUI screen that constitutes a GUI (Graphical User Interface), and the like based on the control of the CPU 201. The CPU 201 controls each part of the display control device 200 so as to generate a display control signal according to a program, generate a video signal to be displayed on the display 205, and output the video signal to the display 205. The display 205 displays an image based on the generated / output image signal. The display control device 200 itself may include an interface for outputting a video signal to be displayed on the display 205, and the display 205 may be configured by an external monitor (television, HMD, etc.).

The operation unit 206 is an input device for receiving user operations, including a character information input device such as a keyboard, a pointing device such as a mouse and a touch panel, a button, a dial, a joystick, a touch sensor, and a touch pad. In the present embodiment, the operation unit 206 includes a touch panel 206a, operation units 206b, 206c, 206d, 206e.

A recording medium 208 such as a memory card, a CD, or a DVD can be attached to and detached from the recording medium I / F 207. The recording medium I / F 207 reads data from the mounted recording medium 208 and writes data to the recording medium 208 based on the control of the CPU 201. The recording medium 208 stores data such as an image to be displayed on the display 205. The external I / F 209 is an interface for connecting to an external device by a wired cable (USB cable or the like) or wirelessly to input / output (data communication) a video signal or an audio signal. The communication I / F 210 is an interface for communicating (wireless communication) with an external device, the Internet 211, or the like to send / receive various data such as files and commands (data communication).

The voice output unit 212 outputs voices of moving images and music data (contents) to be played by the display control device 200, operation sounds, ringtones, various notification sounds, and the like. It is assumed that the audio output unit 212 includes an audio output terminal 212a and a speaker 212b for connecting an earphone or the like, but the audio output unit 212 may output audio data to an external speaker by wireless communication or the like.

The posture detection unit 213 detects the posture (tilt) of the display control device 200 with respect to the gravity direction and the posture of the display control device 200 with respect to each axis in the yaw direction, the pitch direction, and the roll direction, and notifies the CPU 201 of the posture information. Based on the posture detected by the posture detection unit 213, whether the display control device 200 is held horizontally, vertically, upwards, downwards, or in an oblique posture. Etc. can be discriminated. Further, it is possible to determine whether or not the display control device 200 is tilted or large in the rotation direction such as the yaw direction, the pitch direction, and the roll direction, and whether the display control device 200 is rotated in the rotation direction. One sensor such as an acceleration sensor, a gyro sensor, a geomagnetic sensor, an orientation sensor, an altitude sensor, or a combination of a plurality of sensors can be used as the attitude detection unit 213.

As described above, the operation unit 206 includes the touch panel 206a. The touch panel 206a is an input device that is superposed on the display 205 and is configured in a plane so that coordinate information corresponding to the contacted position is output. The CPU 201 can detect the following operations or states on the touch panel 206a.
-A finger or pen that has not touched the touch panel 206a newly touches the touch panel 206a, that is, the start of touch (hereinafter referred to as touch-down).
-A state in which the touch panel 206a is touched by a finger or a pen (hereinafter referred to as touch-on).
-The finger or pen is moving while touching the touch panel 206a (hereinafter referred to as Touch-Move).
-The finger or pen that was touching the touch panel 206a has moved away from the touch panel 206a, that is, the end of the touch (hereinafter referred to as touch-up).
-A state in which nothing is touched on the touch panel 206a (hereinafter referred to as touch-off).

When touchdown is detected, touch-on is also detected at the same time. After touchdown, touch-on usually continues to be detected unless touch-up is detected. When a touch move is detected, a touch-on is detected at the same time. Even if touch-on is detected, touch move is not detected unless the touch position is moved. Touch-off is detected when it is detected that all touched fingers and pens have been touched up.

These operations / states and the position coordinates of the finger or pen touching on the touch panel 206a are notified to the CPU 201 through the internal bus, and the CPU 201 performs what kind of operation (touch operation) on the touch panel 206a based on the notified information. ) Was performed. Regarding the touch move, the moving direction of the finger or pen moving on the touch panel 206a can also be determined for each vertical component and horizontal component on the touch panel 206a based on the change in the position coordinates. When it is detected that the touch move is performed over a predetermined distance, it is determined that the slide operation has been performed. The touch move is a movement operation performed by the user on the touch panel 206a. Various processes corresponding to the touch move, which will be described later, can also be performed in response to a drag operation with the mouse, which is also a movement operation.

The operation of quickly moving a finger on the touch panel 206a by a certain distance and then releasing it is called flicking. Flick is, in other words, touch panel 20
6a This is an operation of swiftly tracing the top with a finger. If it is detected that the touch move is performed at a predetermined speed or more over a predetermined distance and the touch-up is detected as it is, it can be determined that the flick has been performed (it can be determined that there was a flick following the slide operation).

Further, a touch operation in which a plurality of points (for example, two points) are touched at the same time to bring the touch positions closer to each other is called a pinch-in, and a touch operation to move the touch positions away from each other is called a pinch-out. Pinch out and pinch in are collectively called pinch operation (or simply pinch). The touch panel 206a may be any of various types of touch panels such as a resistive film method, a capacitance method, a surface acoustic wave method, an infrared method, an electromagnetic induction method, an image recognition method, and an optical sensor method. good. There are a method of detecting that there is a touch due to contact with the touch panel and a method of detecting that there is a touch due to the approach of a finger or a pen to the touch panel, but any method may be used.

FIG. 2C is an external view of the VR goggles (head mount adapter) 230 to which the display control device 200 can be mounted. The display control device 200 can also be used as a head-mounted display by being attached to the VR goggles 230. The insertion port 231 is an insertion port for inserting the display control device 200. The entire display control device 200 can be inserted into the VR goggles 230 with the display surface of the display 205 facing the headband 232 side (that is, the user side) for fixing the VR goggles 230 to the user's head. The user can visually recognize the display 205 with the VR goggles 230 equipped with the display control device 200 attached to the head without holding the display control device 200 by hand. In this case, when the user moves the head or the entire body, the posture of the display control device 200 also changes. The posture detection unit 213 detects the posture change of the display control device 200 at this time, and the CPU 201 performs a process for VR display based on the posture change. In this case, detecting the posture of the display control device 200 by the posture detecting unit 213 is equivalent to detecting the posture of the user's head (the direction in which the user's line of sight is facing). The display control device 200 itself may be an HMD that can reach the head without VR goggles.

(Update of reference position)
The reference position update process in the display 205 (hereinafter, also referred to as the display unit 205) of the display control device 200 will be described with reference to FIG. The reference position is a position set as a reference for the VR image. The reference position at the start of reproduction of the VR image can be, for example, the position in front of the photographing lens 103a at the time of photographing the VR image.

The reference direction is the posture of the display unit 205 (the posture of the display control device 200) in the initial state. The initial state is the state when the power of the display control device 200 is turned on and the reproduction of the VR image is started. In the initial state, the reference direction is directed to the reference position of the VR image.

The display unit 205 displays a part of the range of the VR image. The range of the display image displayed on the display unit 205 is a range of the VR image including a position (display position) away from the reference position according to the difference between the posture of the display unit 205 and the reference direction. The display position is a reference direction of the display image displayed on the display unit 205, and is, for example, the position of the center of the display image.

In the present embodiment, when the posture of the display unit 205 is maintained in a fixed direction for a predetermined time, that is, when the user gazes at a fixed direction for a predetermined time, the reference position in the VR image is in the fixed direction. Updated to the corresponding position. 3A to 3D are diagrams for explaining the update of the reference position according to the change in the posture of the display unit 205 (hereinafter, also referred to as the posture change) in the reproduction state of the VR image.

FIG. 3A is a diagram illustrating a state of the posture of the display unit 205 at the start of reproduction of the VR image. The default reference position 301 is a reference position at the start of reproduction of the VR image, and is, for example, a front position when the VR image is taken. The reference direction 302 is the posture of the display unit 205 in the initial state. The display position 303 is a position that serves as a reference for a part of a range (display area) of the VR image displayed on the display unit 205 of the display control device 200.

Information indicating the display position such as the azimuth angle and the elevation angle with respect to the reference direction is stored in the memory 202 or the like. At the start of playback of the VR image, the display position 303 is set in the direction of the default reference position 301. That is, the information indicating the display position at the start of reproduction of the VR image is set based on the default reference position 301 and the reference direction 302. The display image 304 is an image in a range displayed based on the information indicating the display position at the start of reproduction of the VR image.

FIG. 3B updates the display position 303 when the posture of the display unit 205 is directed to a fixed time (first predetermined time) different from the reference direction 302 during playback of the display image 304. It is a figure which shows an example. When the posture of the display unit 205 faces a direction different from the reference direction 302, the display unit 205 displays the display image 305 at the position corresponding to the current posture. When the display unit 205 continues a posture different from the reference direction 302 for a predetermined time, the information indicating the display position is updated based on the postures of the display position 303 and the display unit 205 after the posture change. In this case, the default reference position 301 and the reference direction 302 have not changed.

FIG. 3C is a diagram illustrating a reference position update process. When the display position 303 is changed as described with reference to FIG. 3B, the reference position of the VR image is displayed in the direction of the posture of the display unit 205 over a predetermined time (second predetermined time). It is gradually changed to the displayed image 305. That is, until the reference position is changed to the display image 305, the display range displayed on the display unit 205 scrolls in the direction opposite to the reference direction 302. Since the user tries to gaze at the display image 305, the posture of the display unit 205 moves to the reference direction 302 as indicated by the arrow 306. In this way, when the user gazes at a range in a certain direction different from the reference direction 302 for a predetermined time, the gaze constant direction is set as the display position. Then, when the user's posture returns to the reference direction 302, the image of the gaze display position is displayed in the reference direction 302. The reference position update process described with reference to FIG. 3C will be described in detail with reference to the flowchart of FIG.

FIG. 3D is a diagram illustrating another example of the reference position update process. As described with reference to FIG. 3B, when the posture of the display unit 205 is directed toward the display image 305 for a predetermined time, the display position 303 is locked to the display image 305. The display processing of the VR image according to the posture change of the display unit 205 is temporarily stopped. While the posture of the display unit 205 changes to the reference direction 302 as indicated by the arrow 307, the display unit 205 displays the display image 305 regardless of the posture of the display unit 205. When the posture of the display unit 205 moves to the reference direction 302, the display processing of the VR image corresponding to the change in the posture of the display unit 205 is restarted. The reference position update process described with reference to FIG. 3D will be described in detail with reference to the flowchart of FIG.

(Flow of VR view display processing)
FIG. 4 is a flowchart illustrating the VR view display process. This process is realized by expanding the program recorded in the non-volatile memory 203 into the memory 202 and executing it by the CPU 201. This process starts when the power button 206b of the display control device 200 is operated and the power is turned on. When the power is turned on, the CPU 201 initializes the flags, control variables, and the like, and starts the VR view display process.

In S401, the CPU 201 acquires a VR image to be displayed from a communication destination (for example, a digital camera 100) via the recording medium 208 or the communication I / F 210. Further, the CPU 201 acquires information indicating a video range (effective video range) attached to the VR image as attribute information. The information indicating the effective image range is information regarding the range in which the VR image has an effective image in each of the vertical direction and the horizontal direction. The information indicating the effective image range may be any information that can specify the effective image range of the VR image, and may be position information such as an angle of view, a viewing angle, an azimuth angle, an elevation angle, the number of left and right pixels, or coordinates. .. Further, the information indicating the effective image range may be the model information of the camera that captured the VR image. By specifying the model, the CPU 201 can specify the shooting range. Further, the information indicating the effective image range may be zoom information at the time of shooting or the like.

When the CPU 201 acquires the information indicating the effective video range, the CPU 201 can specify (calculate) the invalid video range (non-video range) from the difference from 360 degrees in each of the vertical direction and the horizontal direction. The CPU 201 may acquire information indicating the invalid video range from the attribute information of the VR image and calculate the effective video range from the information indicating the invalid video range. Further, the CPU 201 may acquire the information indicating the effective video range and the information indicating the invalid video range from the attribute information of the VR image.

In S402, the CPU 201 acquires the information of the default reference position 301 from the VR image acquired in S401. The default reference position 301 is the front orientation when the VR image is taken, and is information on the default position that serves as a reference for the position where the VR image is displayed. The information of the default reference position 301 is, for example, information on the horizontal position in the front direction, the vertical direction, or the directional position at the time of shooting of the photographing lens 103a of the digital camera 100.

In S403, the CPU 201 acquires information on the reference direction 302, which is the posture of the display unit 205 in the initial state. The information of the reference direction 302 can be acquired by the posture detection unit 213 detecting the posture of the display unit 205 in the initial state.

In S404, the CPU 201 sets the display position 303 when displaying the VR image on the display unit 205. Specifically, the CPU 201 sets information such as an azimuth angle and an elevation angle with respect to the reference direction 302 as information indicating a display position. In S404, the display unit 205 is directed to the reference direction 302, and the CPU 201 sets information indicating the display position based on the default reference position 301 acquired in S402 and the reference direction 302 acquired in S403.

In S405, the CPU 201 initializes the timer. In S406, the CPU 201 performs the display processing of the VR image acquired in S401. The original image (image data) of the VR image is, for example, a distorted image using equirectangular projection, and is an image in a format in which the position of each pixel can be associated with the coordinates of the surface of the sphere. The CPU 201 maps the original image of this VR image to a sphere, cuts out a part of the image, and displays it. The image displayed in S406 is an image obtained by cutting out a part of the original image and enlarging it, and is an image in which distortion of the original image is removed (or reduced).

In S406, the CPU 201 displays the display image 304 at the display position 303 of the VR image in the reference direction 302 of the display unit 205 based on the information indicating the display position set in S404. That is, the display image 304 of the default reference position 301 is displayed in the reference direction 302 of the display control device 200.

Further, in S406, when the posture detection unit 213 detects a change in the posture of the display unit 205, the CPU 201 displays the display image 305, which is the visual field range of the posture currently facing, in response to the change in the posture of the display unit 205.

In S407, the CPU 201 detects the posture of the display unit 205 by the posture detection unit 213. In S408, the CPU 201 determines whether or not the posture of the display unit 205 detected by the posture detection unit 213 has changed since the timer was initialized in S405. If there is a change in the posture of the display unit 205, the process returns to S405, and if there is no change, the process proceeds to S409.

In S409, the CPU 201 determines whether or not a predetermined time (first predetermined time) has elapsed since the timer was initialized in S405. If a predetermined time has elapsed from the initialization of the timer, the process proceeds to S410, and if not, the process proceeds to S413. The predetermined time is a time of a length that is assumed that the posture of the display unit 205 is maintained in a certain direction by the user watching in a certain direction, for example, 5 seconds or 8 seconds.

In S410, the CPU 201 determines whether or not the current posture of the display unit 205 detected by the posture detection unit 213 is directed to the reference direction 302. The posture of the display unit 205 directed to the reference direction 302 is not limited to the case where the posture of the display unit 205 is in the same direction as the reference direction 302, and the difference in angle from the reference direction 302 is within a predetermined range. It may be the following cases. If the posture of the display unit 205 is directed to the reference direction 302, the process returns to S405, and if it is not directed to the reference direction 302, the process proceeds to S411.

In S411, the CPU 201 updates the information indicating the display position with information such as the azimuth angle and the elevation angle of the current posture of the display unit 205. In S412, the CPU 201 performs a reference position update process for updating the reference position of the VR image to the display position corresponding to the current posture. Details of the processing of S412 will be described later with reference to FIGS. 5 and 6. After the reference position update process is completed, the process returns to S405.

In S413, the CPU 201 updates the timer time to the elapsed time since the timer was initialized in S405. In S414, the CPU 201 determines whether or not there has been any other operation. Other operations include, for example, changing the display mode of the VR image or changing various settings. If there is another operation, the process proceeds to S415, and if there is no other operation, the process proceeds to S416. In S415, the CPU 201 performs processing according to other operations.

In S416, the CPU 201 determines whether or not the operation unit 206 has been operated to end the VR view display. When there is an operation to end the VR view display, the CPU 201 terminates the VR view display application and ends the VR view display process. If there is no end operation of the VR view display for the operation unit 206, the process returns to S407, and the VR view display process is continued.

(Flow of reference position update process)
FIG. 5 is a flowchart illustrating the reference position update process. The reference position update process is a process of updating (changing) the reference position of the VR image to the display position 303 updated to the current posture in S411. The reference position update process shown in FIG. 5 is a detail of the process of S412 in FIG. This process is realized by expanding the program recorded in the non-volatile memory 203 into the memory 202 and executing it by the CPU 201.

In S501, the CPU 201 limits the display processing of the VR image according to the posture change of the display unit 205 detected by the posture detection unit 213. That is, among the VR image display processing corresponding to the posture change of the display unit 205, the display processing for the posture change accompanying the change (update) of the reference position is limited.

In S502, the CPU 201 is based on the information indicating the display position updated in S411.
The process of changing the reference position of the VR image to the display position 303 of the display image 305 displayed on the display unit 205 is started. While the reference position is being changed, the VR image display processing according to the posture change of the display unit 205 is restricted.

In S503, the CPU 201 acquires the posture information of the display unit 205 detected by the posture detection unit 213. In S504, the CPU 201 determines whether or not the posture of the display unit 205 has changed. Here, after the change of the reference position is started in S502, it is determined whether or not the posture change of the display unit 205 is detected in addition to the posture change accompanying the change of the reference position. If a posture change other than the posture change due to the change of the reference position is detected in the posture of the display unit 205, the process proceeds to S505, and if no posture change other than the posture change due to the change of the reference position is detected, the process proceeds to S505. Proceed to S510.

In S505, the CPU 201 suspends the change of the reference position started in S502. In S506, the CPU 201 releases the restriction on the display processing of the VR image according to the change in the posture of the display unit 205. In S507, the CPU 201 performs display processing of the display image according to the posture change of the display unit 205 detected by the posture detection unit 213.

In S508, the CPU 201 acquires the posture information of the display unit 205 detected by the posture detection unit 213. In S509, the CPU 201 determines whether or not the posture of the display unit 205 has changed after the display processing of the VR image corresponding to the change in the posture of the display unit 205 is restricted in S506. If there is a change in the posture of the display unit 205, the process returns to S507, and if there is no change in the posture of the display unit 205, the process returns to S501. In this way, the CPU 201 executes the VR image display process according to the change in the posture of the display unit 205 until the posture of the display unit 205 does not change while the change of the reference position is temporarily stopped. When the posture of the display unit 205 comes to rest, the change of the reference position is restarted in S502.

In S510, the CPU 201 determines whether or not the change of the reference position started in S502 is completed. If the change of the reference position is not completed, the process returns to S503, and if the change of the reference position is completed, the process proceeds to S511. In S511, the CPU 201 releases the restriction of the VR image display processing according to the posture change of the display unit 205, and ends the reference position update processing shown in FIG.

In the present embodiment, when a posture change is detected while changing the reference position of the VR image to the display position, the display control device 200 suspends the change of the reference position and responds to the posture change of the display unit 205. The restriction on the display processing of the VR image is released. The display control device 200 restarts the change of the reference position of the VR image again after the posture change of the display unit 205 disappears, but the present invention is not limited to this. The display control device 200 may operate the change of the reference position and the display processing of the VR image according to the change of the posture of the display unit 205 in parallel without pausing the change of the reference position by the detection of the posture change. .. In this case, the VR image can be displayed without discomfort by relatively updating the amount of change in the reference position and the amount of change in posture. Further, the display control device 200 changes the reference position of the VR image to the display position in a certain direction that the user gazes at over a predetermined time (second predetermined time), but immediately updates the reference position to the display position. You may do so.

(Modification example)
FIG. 6 is a flowchart illustrating the reference position update process according to the modified example. In the reference position update process according to the modification, when the posture of the display unit 205 is directed in a certain direction for a predetermined time, the display position 303 is locked to the display image 305 in the certain direction. Until the posture of the display unit 205 returns to the reference direction 302, the display unit 205 displays the display image 305 regardless of the change in the posture. When the posture of the display unit 205 returns to the reference direction 302, the lock on the display image 305 is released. Further, the reference position of the VR image is updated to the display position 303 of the display image 305.

The reference position update process shown in FIG. 6 is a detail of the process of S412 in FIG. This process is realized by expanding the program recorded in the non-volatile memory 203 into the memory 202 and executing it by the CPU 201.

In S601, the CPU 201 stops the display processing of the VR image according to the posture change of the display unit 205 detected by the posture detection unit 213. Further, the display position 303 updated in S411 is locked to the display image 305. The display unit 205 displays the display image 305 until the posture of the display unit 205 moves in the reference direction 302.

In S602, the CPU 201 determines whether or not the current posture of the display unit 205 is directed to the reference direction 302. The posture of the display unit 205 directed to the reference direction 302 is not limited to the case where the posture of the display unit 205 is in the same direction as the reference direction 302, and the difference in angle from the reference direction 302 is within a predetermined range. It may be the following cases. If the display position 303 is directed to the reference direction 302, the process proceeds to S603, and if it is not directed to the reference direction 302, the process of S602 is repeated.

In S603, the CPU 201 restarts the VR image display process according to the posture change of the display unit 205, and ends the reference position update process. In the reference position update process according to the modified example, when the user gazes at a certain direction for a predetermined time, the display position 303 is locked to the display image 305, and the display unit 205 until the user's posture is returned to the reference direction 302. During that time, the display image 305 is displayed regardless of the change in posture. Therefore, the user can return the posture to the reference direction 302 without losing sight of the display image 305.

In the above embodiment, an example of displaying a VR image on the display control device 200 has been described, but the present invention is not limited to this. The present invention is also applicable to a device for displaying a part of an entire image.

Further, the above-mentioned various controls described as those performed by the CPU 201 may be performed by one hardware, or by sharing the processing by a plurality of hardware (for example, a plurality of processors and circuits), the entire device may be performed. Control may be performed.

Further, although the present invention has been described in detail based on the preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various embodiments within the scope of the gist of the present invention are also included in the present invention. included. Furthermore, each of the above-described embodiments is merely an embodiment of the present invention, and each embodiment can be combined as appropriate.

Further, in the above-described embodiment, the case where the present invention is applied to a display control device (electronic device) has been described as an example, but this is not limited to this example, and any device capable of displaying a VR image can be applied. It is possible. For example, the present invention can be applied to personal computers, PDAs, mobile phone terminals, portable image viewers, digital photo frames, music players, game consoles, electronic book readers, video players, and the like. The present invention is also applicable to wearable devices such as head-mounted displays (HMDs), display devices (including projection devices), tablet terminals, smartphones, AI speakers, home appliances, in-vehicle devices, medical devices, and the like.
<Other Embodiments>

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or recording medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

200: Display control device (electronic device), 201: CPU
205: Display (display unit), 213: Posture detection unit

Claims (15)

A detection means for detecting the posture of the display unit and
A display means for displaying a range of a part of the image including a position separated from the reference position set in the image according to the difference between the posture of the display unit and the reference direction on the display unit.
It has an updating means for updating the reference position to a position corresponding to the certain direction when the state in which the posture of the display unit is directed to a certain direction different from the reference direction for a first predetermined time is maintained. An electronic device characterized by that. The electronic device according to claim 1, wherein the updating means gradually changes the reference position of the image to a position corresponding to the fixed direction over a second predetermined time. The electronic device according to claim 2, wherein the display means limits the display processing of the image according to the posture of the display unit while the reference position is changed. The updating means is characterized in that, when a change in the posture of the display unit other than the change in the posture accompanying the change in the reference position is detected during the change in the reference position, the change in the reference position is temporarily stopped. The electronic device according to claim 2 or 3. The electronic device according to claim 4, wherein the updating means restarts the change of the reference position when the posture of the display unit is stationary during the suspension of the change of the reference position. The update means according to any one of claims 3 to 5, wherein when the change of the reference position is completed, the restriction of the display processing of the image according to the posture of the display unit is released. Electronic equipment. The update means according to claims 2 to 6, wherein the updating means controls the display process based on the posture change of the display unit other than the posture change accompanying the change of the reference position among the posture changes of the display unit. The electronic device according to any one item. The electronic device according to any one of claims 2 to 7, wherein the updating means uses the posture of the display unit as a reference direction when the change of the reference position is completed. The update means according to claim 1, wherein when the posture of the display unit is in the same direction as the reference direction, the display process of the image according to the posture of the display unit is restarted. Electronics. The electronic device according to any one of claims 1 to 9, wherein the image is an omnidirectional image. The electronic device according to any one of claims 1 to 10, wherein the image includes information on the reference position. The electronic device according to any one of claims 1 to 11, wherein the posture of the display unit at the start of reproduction of the image is set to the reference direction during reproduction of the image. A detection step that detects the posture of the display unit,
A display step of displaying a range of a part of the image including a position separated from the reference position set in the image according to the difference between the posture and the reference direction of the display unit on the display unit.
An update step of updating the reference position to a position corresponding to the constant direction when the state in which the posture of the display unit is directed to a certain direction different from the reference direction for a first predetermined time is maintained.
A method for controlling an electronic device, which comprises. A program for causing a computer to function as each means of the electronic device according to any one of claims 1 to 12. A computer-readable recording medium containing a program for causing the computer to function as each means of the electronic device according to any one of claims 1 to 12.

Images