JP7271296B2 - IMAGING DEVICE, IMAGING DEVICE CONTROL METHOD AND PROGRAM - Google Patents

Description

The present invention relates to an imaging device, an imaging device control method, and a program.

2. Description of the Related Art In recent years, imaging apparatuses capable of capturing an image in a wider range than the human viewing angle, such as an omnidirectional image (omnidirectional image), have become widespread. Such an imaging apparatus simultaneously drives a plurality of camera units to capture subject images.

In Japanese Patent Application Laid-Open No. 2004-100001, power saving is achieved without impairing the function of simultaneously capturing 360-degree images in all directions by lowering the frequency of drive pulses in some of a plurality of camera units according to the presence or absence of an operation instruction. We are trying to

In Japanese Patent Application Laid-Open No. 2002-200001, when there is no change in the state of an imaging device and a remote controller that operates the imaging device (operating state or stationary state acquired by an acceleration sensor) for a predetermined period of time, the power of the imaging device is turned off. Efforts are being made to save power.

Japanese Unexamined Patent Application Publication No. 2010-232908 JP 2017-139812 A

However, in Patent Document 1, when displaying a part of an omnidirectional image of a live view image (an image obtained from a camera in substantially real time), a camera unit (hereinafter referred to as an effective camera) for capturing an image displayed on a display device The output of the camera section other than the camera section is wasted. In addition, in Patent Document 2, since the output of the camera section is maintained while the imaging device or the display device is being moved, similarly to the above, the output of the camera sections other than the effective camera section is wasted.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide a technique for appropriately suppressing power consumption of each imaging device when combining images output from a plurality of imaging devices and displaying the synthesized images on a display device.

A first aspect of the present invention is
a plurality of imaging processing means for outputting images obtained by imaging subjects in different directions;
generating means for generating a synthesized image by synthesizing the images output by the plurality of imaging processing means;
a recording means for recording the composite image on a recording medium;
Acquisition means for acquiring display range information indicating a display range of the composite image to be displayed on a screen;
while the composite image is being recorded on the recording medium, controlling the plurality of imaging processing means to be driven in the first mode regardless of the display range information;
while the composite image is not being recorded on the recording medium, the imaging processing means for outputting an image corresponding to the display range of the composite image is driven in a first mode based on the display range information. a control means for controlling the imaging processing means for outputting an image corresponding to a range not displayed in the synthesized image so as to be driven in a second mode which is more power-saving than the first mode;
An imaging device characterized by having

A second aspect of the present invention is
A control method for an imaging device having a plurality of imaging processing means for outputting images obtained by imaging subjects in different directions,
a generating step of generating a synthesized image by synthesizing the images output by the plurality of imaging processing means;
a recording step of recording the composite image on a recording medium;
an obtaining step of obtaining display range information indicating a display range of the composite image displayed on a screen;
while the composite image is being recorded on the recording medium, controlling the plurality of imaging processing means to be driven in the first mode regardless of the display range information;
while the composite image is not being recorded on the recording medium, the imaging processing means for outputting an image corresponding to the display range of the composite image is driven in a first mode based on the display range information. a control step of controlling the imaging processing means for outputting an image corresponding to an area not displayed in the synthesized image so as to be driven in a second mode that saves power compared to the first mode;
A control method for an imaging device characterized by comprising:

A third aspect of the present invention is a computer program for causing a computer to execute each step of the above control method.

ADVANTAGE OF THE INVENTION According to this invention, when combining the image output from several imaging devices and displaying on a display apparatus, the technique which suppresses the power consumption of each imaging device appropriately can be provided.

1 is a front external view showing an example of an imaging device according to an embodiment; FIG. 1 is a rear external view showing an example of an imaging device according to an embodiment; FIG. 1 is a functional block diagram showing an example of an imaging device according to an embodiment; FIG. 1 is an external view showing an example of a display control device according to an embodiment; FIG. 1 is a functional block diagram showing an example of a display control device according to an embodiment; FIG. FIG. 4 is a diagram showing an example of a synthesized image according to the embodiment; FIG. 4 is a diagram showing an example of part of a synthesized image according to the embodiment;

4 is a flowchart showing an example of processing of an imaging device according to an embodiment; 4 is a flow chart showing an example of processing according to a command of the imaging device according to the embodiment; 4 is a flowchart showing an example of shooting processing of the imaging device according to the embodiment; 4 is a flowchart showing an example of processing of the display control device according to the embodiment; 4 is a flowchart showing an example of display end processing of the display control device according to the embodiment; 4 is a flowchart showing an example of imaging processing of the display control device according to the embodiment; 4 is a flowchart showing an example of display change processing of the display control device according to the embodiment; 4 is a flowchart showing an example of power saving processing of the display control device according to the embodiment; 4 is a flowchart showing an example of boundary processing of the display control device according to the embodiment; 4 is a flowchart showing an example of boundary processing of the display control device according to the embodiment;

Flowchart showing an example of a target range of startup time reduction processing according to the embodiment Flowchart showing an example of startup time reduction processing according to the embodiment A diagram showing an example of a no-operation time setting screen according to the embodiment. 3 is a flow chart showing an example of a no-operation time setting process according to an embodiment;

4 is a flowchart showing an example of processing of an imaging device according to an embodiment; 4 is a flowchart showing an example of processing of the display control device according to the embodiment; 4 is a flowchart showing an example of processing of an imaging device according to an embodiment; 4 is a flowchart showing an example of processing of the display control device according to the embodiment;

(First embodiment)
Embodiments of the present invention will be described below.

<Overview>
In this embodiment, an example will be described in which an imaging device having a plurality of camera units is used to capture an omnidirectional image, and a display control device is used to display a portion of the omnidirectional image. The omnidirectional image in this embodiment is an image having a field of view of 360 degrees obtained by combining image data (captured data) captured by a plurality of camera units of an imaging device. The configurations and processing contents of the imaging device 100 and the display control device 200 according to this embodiment will be described in order below.

<Configuration of Imaging Device 100>
FIG. 1A is a front external view (perspective view) showing an example of an imaging device 100 according to this embodiment. FIG. 1B is a rear external view (perspective view) of the imaging device 100. FIG. The imaging device 100 is an information processing device (computer) including an arithmetic device (processor), a memory, a storage device, an input/output device, and the like. The imaging device 100 executes a program stored in the storage device, thereby providing functions of the imaging device 100, which will be described later. Some or all of those functions may be implemented by dedicated logic circuits such as ASICs and FPGAs. In this embodiment, an imaging apparatus having two camera units will be described as an example, but the number of camera units may be three or more.

The imaging device 100 according to the present embodiment can also be regarded as an omnidirectional camera (omnidirectional camera) having two camera units a and b. The camera unit a is a wide-angle camera that covers a wide range of 180 degrees or more up, down, left, and right on the front side of the imaging device 100 . The camera unit b is a wide-angle camera with a wide range of 180 degrees or more on the back side of the image pickup device 100 for photographing. Barriers 102a and 102b are protective windows of taking lenses 103a and 103b for camera parts a and b. The barriers 102a and 102b may be the outer surfaces of the imaging lenses 103a and 103b themselves.

The display unit 28 is a display unit that displays various information. The shutter button 61 is an operation unit for instructing shooting. The mode changeover switch 60 is an operation unit for switching between various modes. The connection I/F 25 is a connector between the imaging device 100 and a connection cable for connecting to external devices such as smartphones, personal computers, and televisions. The operation unit 70 is an operation unit including operation members such as various switches, buttons, dials, and touch sensors for receiving various operations from the user. The power switch 72 is a push button for switching ON/OFF of the power.

The light-emitting unit 21 is a light-emitting member such as a light-emitting diode (LED), and notifies the user of various states of the imaging device 100 using light-emitting patterns and light-emitting colors. The fixing part 40 is, for example, a tripod screw hole, and is a member for fixing to a fixing device such as a tripod.

FIG. 1C is a functional block diagram showing an example of the imaging device 100. As shown in FIG.

The barriers 102a and 102b cover the imaging systems of the camera units a and b including the imaging lenses 103a and 103b of the imaging apparatus 100, thereby allowing the imaging systems including the imaging lenses 103a and 103b, the shutters 101a and 101b, and the imaging units 22a and 22b. to prevent dirt and damage. The imaging lenses 103a and 103b are a lens group including a zoom lens and a focus lens, and are, for example, wide-angle lenses. The shutters 101a and 101b are shutters having a diaphragm function for adjusting the amount of subject light incident on the imaging units 22a and 22b. The image pickup units 22a and 22b are image pickup devices configured by CCDs, CMOS devices, or the like that convert optical images into electrical signals. The A/D converters 23a and 23b are functional units that convert analog signals output from the imaging units 22a and 22b into digital signals.

In this embodiment, VR (Virtual Reality) images are captured by the imaging units 22a and 22b. VR images include omnidirectional images (omnidirectional images) captured by an omnidirectional camera, and panoramic images with an image range (effective image range) that is wider than the display range that can be displayed on the display device at once. (panorama image data), etc. VR images include not only still images but also moving images and live view (LV: Live View) images (images obtained from a camera in almost real time).

Here, the VR image is a video range (effective video range). In addition, even if the VR image is less than 360 degrees vertically and less than 360 degrees horizontally, it can be displayed at a wide angle of view (viewing range) that is wider than the angle of view that can be taken with a normal camera, or on a display device at once. Images with a video range (effective video range) wider than the display range are also included. For example, an image captured by an omnidirectional camera capable of capturing a subject with a field of view (angle of view) of 360 degrees in the horizontal direction (horizontal angle, azimuth angle) and a vertical angle of 210 degrees centered on the zenith is It is a kind of VR image.

Also, for example, an image captured by a camera capable of capturing a subject with a field of view (angle of view) of 180 degrees in the horizontal direction (horizontal angle, azimuth angle) and 180 degrees in the vertical angle centered on the horizontal direction is a VR image. It is one kind. That is, an image that has a visual range of 160 degrees (±80 degrees) or more in the vertical direction and the horizontal direction and has a video range that is wider than the range that can be visually recognized by humans at once is a VR image. A type of image. When this VR image is displayed in VR, which will be described later, by changing the orientation of the display device in the left-right rotation direction, seamless omnidirectional video can be viewed in the left-right direction (horizontal rotation direction). In the vertical direction (vertical rotation direction), it is possible to view seamless omnidirectional images within a range of ±105 degrees when viewed from directly above (zenith). It becomes a blank area where no image exists. A VR image can also be said to be "an image whose video range is at least part of a virtual space (VR space)".

The VR display is a display method capable of changing the display range, which displays an image within the viewing range corresponding to the posture of the display device among the VR images. For example, when a head-mounted display (HMD) is worn as a display device for viewing, an image within a visual field corresponding to the orientation of the user's face is displayed. For example, in the VR image, at a certain point in time, the viewing angle (angle of view) is centered at 0 degrees in the horizontal direction (a specific direction, such as north) and 90 degrees in the vertical direction (90 degrees from the zenith, that is, horizontal). can be displayed. From this state, if the posture of the display device is reversed (changed from south to north, etc.), the same VR image will be 180 degrees in the left and right direction (reverse orientation, for example, south) and up and down. The display range is changed to an image with a viewing angle centered at 90 degrees (horizontal).

Assuming that the user is viewing the HMD, if the user turns his/her face from north to south (that is, turns his/her back), the image displayed on the HMD also changes from the north image to the south image. With such VR display, it is possible to provide the user with the sensation of being visually present in the VR image (inside the VR space). Also, the above-described VR display may be performed by attaching a display device such as a smartphone to VR goggles (head mount adapter). In this case, the attached smart phone can be said to be a kind of HMD. Note that the display method of the VR image is not limited to the above, and for example, the display range may be moved (scrolled) according to the user's operation on the touch panel or direction buttons instead of the change in posture.

The image processing unit 24 is a functional unit that performs resizing processing such as predetermined pixel interpolation and reduction, and color conversion processing on data from the A/D converters 23 a and 23 b or data from the memory control unit 15 . Further, the image processing unit 24 performs predetermined arithmetic processing using the captured image data. Based on the calculation result obtained by the image processing section 24, the system control section 50 performs exposure control and distance measurement control. As a result, TTL (Through The Lens) AF (Auto Focus) processing, AE (Auto Exposure) processing, and EF (Electronic Flash) processing are performed. Further, the image processing unit 24 performs predetermined arithmetic processing using the captured image data, and performs TTL AWB (Auto White Balance) processing based on the obtained arithmetic result.

The image processing unit 24 performs basic image processing on the two images (fisheye images) obtained from the A/D converters 23a and 23b, and then synthesizes them (connecting image processing) to create a single VR image. Generate an image (composite image). FIG. 2C shows image processing of an image 221 (broken line rectangle) obtained from the A/D converter 23a of the camera section a and an image 222 (broken line rectangle) obtained from the A/D converter 23b of the camera section b. FIG. 10 shows a conceptual diagram of synthesizing the results to generate a VR image 223 (solid-line rectangle). In the stitching image processing of two images, the image processing unit 24 calculates the shift amount between the reference image and the comparison image for each area by pattern matching processing in each of the two images, and detects the stitching position. Then, the image processing unit 24 performs distortion correction processing on each of the two images by geometric transformation in consideration of the detected joint position and the characteristics of each optical system lens, and converts them into an omnidirectional image format. The image processing unit 24 blends the two omnidirectional image format images to finally generate one omnidirectional image data (VR image). The generated omnidirectional image data becomes an image using equirectangular projection, and the position of each pixel can be associated with the coordinates of the surface of the sphere.

Output data from the A/D converters 23a and 23b are written to the memory 32 via the image processing section 24 and the memory control section 15, or via the memory control section 15. FIG. The memory 32 stores image data obtained by the imaging unit 22 and converted into digital data by the A/D converter 23 and images to be output from the connection I/F 25 to an external display. The memory 32 has a storage capacity sufficient to store a predetermined number of still images, moving images for a predetermined period of time, and audio.

The memory 32 also serves as an image display memory (video memory). Image display data stored in the memory 32 can be output from the connection I/F 25 to an external display. VR images generated by the image processing unit 24 using images captured by at least one of the imaging units 22a and 22b and stored in the memory 32 are sequentially transferred to and displayed on an external display. This allows the external display to function as an electronic viewfinder and perform live view display (LV display). An image displayed in live view is hereinafter referred to as an LV image. Note that the VR images accumulated in the memory 32 are transferred to an external device (smartphone, etc.) wirelessly connected via the communication unit 54, and displayed on the external device side for live view display (remote LV display). may

The nonvolatile memory 56 is a memory as an electrically erasable/recordable recording medium, and for example, EEPROM (Electrically Erasable Programmable Read-Only Memory) or the like is used. The nonvolatile memory 56 stores constants, programs, etc. for the operation of the system control unit 50 . The program here is a computer program for executing various flowcharts described later in this embodiment.

The system control unit 50 is a control unit having at least one processor or circuit, and is a functional unit that controls the imaging device 100 as a whole. The system control unit 50 executes the program recorded in the non-volatile memory 56 described above, thereby realizing each process of this embodiment described later. A RAM, for example, is used for the system memory 52 . In the system memory 52, constants and variables for operation of the system control unit 50, programs read from the nonvolatile memory 56, and the like are expanded. The system control unit 50 also performs display control by controlling the memory 32 , the image processing unit 24 and the memory control unit 15 . The system timer 53 is a timer that measures the time used for various controls and the time of the built-in clock.

A mode switch 60 , a shutter button 61 , and an operation unit 70 are operation units for inputting various operation instructions to the system control unit 50 . The mode changeover switch 60 switches the operation mode of the system control unit 50 between a still image recording mode, a moving image shooting mode, a reproduction mode, a communication connection mode, and the like. Modes included in the still image recording mode include an auto shooting mode, an auto scene determination mode, a manual mode, an aperture priority mode (Av mode), a shutter speed priority mode (Tv mode), and a program AE mode. In addition, there are various scene modes, custom modes, and the like, which are shooting settings for each shooting scene. The user switches directly to one of the above modes using the mode selector switch 60 . Alternatively, the user may select one of the plurality of modes displayed on the display unit 28 after temporarily switching to the shooting mode list screen with the mode switching switch 60, and switch using another operation member. good. Similarly, the movie shooting mode may also include multiple modes.

The power switch 72 is a push button for switching power ON/OFF. The light-emitting unit 21 is a light-emitting member such as a light-emitting diode (LED), and is a functional unit that notifies the user of various states of the imaging device 100 using light-emitting patterns and light-emitting colors. A recording medium 150 is a recording medium such as a memory card or a hard disk.

The first shutter switch 62 is turned on when the shutter button 61 provided in the imaging device 100 is pressed halfway (imaging preparation instruction), and generates a first shutter switch signal SW1. In response to the first shutter switch signal SW1, the system control unit 50 starts photographing preparation operations such as AF processing, AE processing, AWB processing, and EF processing described above.

The second shutter switch 64 is turned ON when the operation of the shutter button 61 is completed, that is, when the shutter button 61 is fully pressed (imaging instruction), and generates a second shutter switch signal SW2. The system control unit 50 starts a series of photographing processing operations from signal reading from the imaging unit 22 to writing of image data in the recording medium 150 in response to the second shutter switch signal SW2.

It should be noted that the shutter button 61 is not limited to one that can be operated in two stages of full-press and half-press, and may be an operation member that can be pressed only in one stage. In that case, the photographing preparation operation and the photographing process are continuously performed in response to one step of pressing. This is the same operation as when the shutter button, which can be half-pressed and fully-pressed, is fully pressed (operation when SW1 and SW2 occur almost simultaneously).

Each operation member of the operation unit 70 is appropriately assigned a function for each scene by selecting and operating various function icons and options displayed on the display unit 28, and acts as various function buttons. The function buttons include, for example, an end button, a return button, an image forward button, a jump button, a refinement button, an attribute change button, and the like. For example, when the menu button is pressed, a menu screen on which various settings can be made is displayed on the display unit 28 . The user can intuitively perform various settings by operating the operation unit 70 while viewing the menu screen displayed on the display unit 28 .

The power control unit 80 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching blocks to be energized, and the like, and detects whether or not a battery is installed, the type of battery, and the remaining amount of the battery. Also, the power control unit 80 controls the DC-DC converter based on the detection result and the instruction from the system control unit 50, and supplies necessary voltage to each unit including the recording medium 150 for a necessary period. The power source unit 30 is a power source including a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, an AC adapter, or the like.

A recording medium I/F 18 is an interface with a recording medium 150 such as a memory card or hard disk. A recording medium 150 is a recording medium such as a memory card for recording captured images, and is composed of a semiconductor memory, an optical disk, a magnetic disk, or the like. The recording medium 150 may be an exchangeable recording medium detachable from the imaging device 100, or may be a built-in recording medium.

The communication unit 54 is connected wirelessly or by a wired cable, and transmits and receives video signals, audio signals, and the like. The communication unit 54 can be connected to a wireless LAN (Local Area Network) or the Internet. The communication unit 54 can also communicate with an external device via Bluetooth (registered trademark) or Bluetooth Low Energy (registered trademark). The communication unit 54 can transmit images (including LV images) captured by the imaging units 22a and 22b and images recorded in the recording medium 150, and receives images and other various information from external devices. may

The orientation detection unit 55 detects the orientation of the imaging device 100 with respect to the direction of gravity. Based on the posture detected by the posture detection unit 55, the system control unit 50 determines whether the image captured by the image capturing unit 22 is an image captured with the image capturing device 100 held horizontally or captured with the image capturing device 100 held vertically. It is possible to determine whether it is an image that has been processed. Also, the system control unit 50 can determine how much the image is tilted in the directions of the three axes of yaw, pitch, and roll. The system control unit 50 adds orientation information corresponding to the posture detected by the posture detection unit 55 to the image file of the VR image captured by the imaging units 22a and 22b, rotates the image (corrects the tilt of the image, and so on). It is possible to adjust the orientation of the image and record it. As the posture detection unit 55, an acceleration sensor, a gyro sensor, a geomagnetic sensor, an azimuth (angle) sensor, an altitude sensor, or the like may be used singly or in combination of two or more. The orientation detection unit 55 can also detect the movement of the imaging device 100 (pan, tilt, lift, whether it is stationary, etc.) using the above sensor.

The microphone 20 is a microphone that collects sounds around the imaging device 100 that are recorded as sounds of moving images of VR images. The connection I/F 25 is a connection plug with an HDMI (registered trademark) cable, a USB cable, or the like, for connecting to an external device and transmitting/receiving video.

<Configuration of Display Control Device 200>
FIG. 2A is an external view showing an example of a display control device 200, which is a type of electronic device. The display control device 200 is an information processing device (computer) including an arithmetic device (processor), a memory, a storage device, an input/output device, and the like. The functions of the display control device 200, which will be described later, are provided by the display control device 200 executing the programs stored in the storage device. Some or all of these functions may be implemented by dedicated logic circuits such as ASICs and FPGAs.

A display 205 is a display unit that displays images and various types of information. In this embodiment, the display 205 is configured integrally with a touch panel 206a as described later, and detects a touch operation on the display surface of the display 205. FIG. Also, the display control device 200 can perform VR display of a VR image (VR content) on the display 205 . The operation unit 206 (operation receiving unit) includes a touch panel 206a and operation units 206b, 206c, 206d, and 206e. The operation unit 206 b is a power button that receives an operation for switching ON/OFF of the power of the display control device 200 . The operation units 206c and 206d are buttons for increasing or decreasing the volume of audio output from an audio output unit (for example, a speaker 212b to be described later). The operation unit 206e is a button for displaying the home screen on the display 205. FIG. The audio output terminal 212a is an earphone jack, and outputs audio to an earphone, an external speaker, or the like. The speaker 212b is a built-in speaker that produces sound.

FIG. 2B is a functional block diagram showing an example of the display control device 200. As shown in FIG. The display control device 200 can be configured using a display device such as a smartphone.

2B, CPU 201, memory 202, nonvolatile memory 203, image processing unit 204, display 205, operation unit 206, storage medium I/F 207, external I/F 209, and communication I/F 210 are connected to internal bus 250. It is Also connected to the internal bus 250 are an audio output unit 212 , an attitude detection unit 213 and a system timer 214 . Each unit connected to the internal bus 250 exchanges data with each other via the internal bus 250 .

The CPU 201 is a control unit that controls the entire display control device 200, and is configured using at least one processor or circuit. The memory 202 is, for example, a RAM (such as a volatile memory using a semiconductor element). The CPU 201 uses the memory 202 as a work memory to control each part of the display control device 200 according to a program stored in the nonvolatile memory 203, for example. The nonvolatile memory 203 stores image data, audio data, other data, various programs for the CPU 201 to operate, and the like. The nonvolatile memory 203 is, for example, flash memory or ROM.

Under the control of the CPU 201, the image processing unit 204 processes images stored in the nonvolatile memory 203 and the storage medium 208, video signals obtained via the external I/F 209, and images obtained via the communication I/F 210. It is a functional unit that performs various image processing on such as. The image may be a still image or a moving image. The image processing performed by the image processing unit 204 includes A/D conversion processing, D/A conversion processing, image data encoding processing, compression processing, decoding processing, enlargement processing/reduction processing (resize), noise reduction processing, and color processing. Conversion processing etc. are included. Various image processing such as panorama expansion, mapping processing, and conversion of a VR image, which is a wide-range image having wide-range data, is also performed on an omnidirectional image or an image that is not omnidirectional. The image processing unit 204 may be composed of a dedicated circuit block for performing specific image processing. Further, depending on the type of image processing, the CPU 201 can perform image processing according to a program without using the image processing unit 204 .

Here, the case where the image processing unit 24 of the imaging device 100 synthesizes (splice image processing) two images (fisheye images) to generate a single VR image (composite image) has been described. The control device 200 may generate a single VR image (composite image). In this case, two images (fisheye images) are transmitted from the imaging device 100 to the display control device 200 without being synthesized. Then, the display control device 200 combines (splice image processing) the two images (fisheye images) received from the imaging device 100 to generate a single VR image (composite image). The generated VR image is recorded on the recording medium 208 from the recording start instruction to the recording end instruction.

The display 205 displays images, GUI screens that form a GUI (Graphical User Interface), and the like, under the control of the CPU 201 . The CPU 201 controls each part of the display control device 200 to generate a display control signal according to a program, generate a video signal to be displayed on the display 205 , and output the video signal to the display 205 . A display 205 displays an image based on the output image signal. The display control device 200 itself includes an interface for outputting a video signal to be displayed on the display 205, and the display 205 may be an external monitor (such as a television).

An operation unit 206 (accepting unit) is an input device for accepting user operations. The operation unit 206 includes, for example, a character information input device such as a keyboard, a pointing device such as a mouse and a touch panel, buttons, dials, joysticks, touch sensors, and touch pads. It should be noted that the touch panel is an input device that is superimposed on the display 205 and configured in a two-dimensional manner so as to output coordinate information according to the touched position. Note that the operation unit 206 may be a receiving device for receiving signals from an external input device such as a remote controller instead of an input device.

A storage medium I/F 207 can be loaded with a storage medium 208 such as a memory card, a CD, or a DVD. I do. The external I/F 209 is an interface for connecting with an external device by a wired cable or wirelessly and inputting/outputting a video signal and an audio signal. The communication I/F 210 is an interface for communicating with an external device, the Internet (NET) 211, etc., and transmitting and receiving various data such as files and commands.

The audio output unit 212 is a functional unit that outputs audio of video and music data, operation sounds, ringtones, various notification sounds, and the like. The audio output unit 212 includes an audio output terminal 212a for connecting an earphone or the like and a speaker 212b, but audio output may be performed by wireless communication or the like.

The posture detection unit 213 is a functional unit that detects the posture of the display control device 200 with respect to the direction of gravity and the tilt of the posture with respect to each of the yaw, roll, and pitch axes. Based on the orientation detected by the orientation detection unit 213, whether the display control device 200 is held horizontally, held vertically, directed upward, downward, or oblique. etc. can be determined. At least one of an acceleration sensor, a gyro sensor, a geomagnetic sensor, a direction sensor, an altitude sensor, and the like can be used as the posture detection unit 213, and a plurality of sensors can be used in combination. A system timer 214 is a timer that measures the time used for various controls and the time of a built-in clock.

In this embodiment, the operation unit 206 includes a touch panel 206a. The CPU 201 detects the following operations or states on the touch panel 206a.
- The touch panel 206a is newly touched by a finger or pen that has not touched the touch panel 206a, that is, the start of touch (hereinafter referred to as Touch-Down).
A state in which a finger or pen is touching the touch panel 206a (hereinafter referred to as Touch-On).
- The finger or pen is moving while touching the touch panel 206a (hereinafter referred to as Touch-Move).
- The finger or pen touching the touch panel 206a is separated from the touch panel 206a, that is, the end of the touch (hereinafter referred to as Touch-Up).
A state in which nothing is touched on the touch panel 206a (hereinafter referred to as Touch-Off).

When touchdown is detected, touchon is also detected at the same time. After touchdown, touchon continues to be detected unless touchup is detected. Touch-on is detected at the same time when touch-move is detected. Even if touch-on is detected, touch-move is not detected if the touch position does not move. Touch-off is detected when it is detected that all the fingers or pens that were in touch have touched up.

The CPU 201 is notified of these operations/states and the coordinates of the position where the finger or pen touches the touch panel 206a through the internal bus. Then, the CPU 201 determines what kind of operation (touch operation) has been performed on the touch panel 206a based on the notified information. As for the touch move, the moving direction of the finger or pen moving on the touch panel 206a is also determined for each vertical component/horizontal component on the touch panel 206a based on the change in the position coordinates. When it is detected that the touch-move has been performed for a predetermined distance or more, the CPU 201 determines that a slide operation has been performed. An operation of touching the touch panel 206a with a finger and quickly moving it by a certain distance and then releasing it is called a flick. A flick is, in other words, an operation of quickly tracing the touch panel 206a as if flicking it with a finger. It is determined that a touch-move has been performed over a predetermined distance at a predetermined speed or more, and if a touch-up is detected as it is, it is determined that a flick has been performed (determined that a flick has occurred following a slide operation). Furthermore, a touch operation of simultaneously touching a plurality of points (for example, two points) to bring the touch positions closer together is called pinch-in, and a touch operation of moving the touch positions away from each other is called pinch-out. Pinch-out and pinch-in are collectively called pinch operation (or simply pinch). The touch panel 206a may be any one of various types of touch panels such as a resistive film type, an electrostatic capacitance type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, an optical sensor type, and the like. good. There is a method of detecting that there is a touch when there is contact with the touch panel, and a method of detecting that there is a touch when there is an approach of a finger or pen to the touch panel.

<Contents of Processing of Imaging Apparatus 100>
3A to 3C are flowcharts showing an example of various processes performed by the imaging device 100. FIG. The various types of processing include shooting processing (imaging processing) by operating the imaging device 100 itself, device control processing by commands received from the display control device 200, and the like. The above processing is realized by loading the program recorded in the nonvolatile memory 56 into the system memory 52 and executing it by the system control unit 50 . Also, the control described in FIG. 3A is executed by the system control unit 50 . When the imaging device 100 is powered on and the communication unit 54 or the connection I/F 25 is enabled, the processing shown in FIG. 3A is started.

First, the system control unit 50 processes images captured by the camera units a and b in the image processing unit 24, and starts outputting the images to the memory 32 (S301). Then, the system control unit 50 determines whether or not the imaging device 100 is communicating with the display control device 200 via the communication unit 54 (S302). If communication is not in progress (S302-NO), the system control unit 50 executes the process of step S312.

If the communication is in progress (S302-YES), the system control unit 50 starts transmitting the real-time VR image stored in the memory 32 as a live image from the communication unit 54 (S303). Next, the system control unit 50 transmits the imaging device status (state) to the display control device 200 (S304). Here, the imaging device status includes ON/OFF of the camera section, ON/OFF of shooting, ON/OFF of recording, ON/OFF of image transmission (video transmission) to the display control device 200, and the like. The imaging device status includes information such as whether power is supplied to the imaging unit 22, whether power is supplied to the A/D converter 23, ON/OFF of image processing, and the like. The ON state of the camera section means that at least the imaging section 22 is supplied with power and the photographing is ON, and the image transmission may be ON or OFF. The OFF state of the camera section means that at least photographing is OFF (image transmission is also OFF), and power supply to the imaging section 22 may or may not be provided. Next, the system control unit 50 determines whether or not live video is being displayed on the display control device 200 (S305). If it is not being displayed (S305-NO), the system control unit 50 executes the process of step S308. Note that the system control unit 50 may determine whether or not the live video is being displayed by acquiring information from the display control device 200 using wireless communication or the like. Further, the system control unit 50 may make the determination in step 305 according to whether live video is being transmitted to the display control device 200 or not.

If live video is being displayed (S305-YES), the system control unit 50 determines whether or not a command has been received from the display control device 200 (S306), and if no command has been received (S306 -NO) executes the process of step S304. If the command has been received (S306-YES), the system control unit 50 executes processing according to the command from the display control device 200 (S307). The details of the processing in step S307 will be described later.

Next, in step S308, the system control unit 50 determines whether or not "the live video is being transmitted to the outside and the communication with the display control device 200 is not being performed". If "live video is being sent to the outside and communication with the display control device 200 is not in progress" (S308-YES), the system control unit 50 stops sending the live video (S309).

Next, in step S<b>310 , the system control unit 50 determines whether or not a shooting instruction has been issued based on the operation of the imaging device 100 . The operation of the imaging device 100 is, for example, an operation by pressing the shutter button 61 . If a shooting instruction has been received (S310-YES), the system control unit 50 executes the shooting process (S311) shown in FIG. 3B. The details of the photographing process in step S311 will be described later.

Next, in step S312, the system control unit 50 determines whether or not a camera stop operation has been performed. If a stop operation has been performed (S312-YES), the system control unit 50 stops recording if recording is in progress (S313-YES) (S314), and then turns off each camera unit (S315). .

<<Process according to the command (S307)>>
FIG. 3B is a flowchart showing details of the processing in step S307. In the present embodiment, an example will be described in which the imaging device 100 receives commands related to image capturing processing, ON/OFF switching processing of the camera unit, and ON/OFF switching processing of image transmission to the display control device 200 . Note that the command received by the imaging device 100 is not limited to the above, and may be, for example, a command to change the orientation of the camera unit or a command to change the state of the imaging device status described above.

Here, the command regarding the shooting process is a command (shooting instruction) performed in the shooting instruction process (S408) of the display control device 200, which will be described later. Also, the commands for turning on the camera unit and turning on the image transmission are commands that are executed in the process of changing the display range of the display control device 200 (S410), which will be described later. Further, the commands for turning off the camera unit and turning off image transmission are commands performed in power saving processing (S413) of the display control device 200, which will be described later.

In addition, the display control device 200 instructs switching between ON/OFF of the camera unit or ON/OFF of image transmission depending on whether the imaging device 100 is recording. This is because when the camera unit is turned on, only image transmission needs to be turned on because the power of the imaging device 100 (power supply to the camera unit) is already on during recording. Also, when the camera unit is turned off (power saving processing), if recording is in progress, only image transmission is turned off while maintaining the recording state, thereby saving the power of the camera unit. Similarly, in the processing of the imaging device 100, the processing is divided depending on whether the imaging device 100 is recording or not. good too. Processing (S307) according to the above command in the imaging apparatus 100 will be described below.

The system control unit 50 determines whether or not the received command is a photographing instruction (S3071). If it is not a shooting instruction (S3071-NO), the system control unit 50 performs the following processing depending on whether recording is in progress. If recording is not in progress (S3072-NO), the system control unit 50 switches ON/OFF of some camera units in accordance with the command from the display control device 200 (S3073). If recording is in progress (S3072-YES), the system control unit 50 transmits a VR image obtained by synthesizing images output from some of the plurality of camera units according to a command from the display control device 200. (Transmission of live video) is switched ON/OFF (S3074). If the received command is a shooting instruction (S3071-YES), the system control unit 50 executes shooting processing according to the command from the display control device 200 (S3075). Here, the shooting instruction in the present embodiment includes a still image shooting instruction or a moving image recording start instruction/recording stop instruction. Note that the imaging process in step S3075 may be the same as the imaging process in step S311. After the processing of steps S3073 to S3075, the system control unit 50 executes the processing of step S304 shown in FIG. 3A.

<<Shooting process (S311)>>
FIG. 3C is a flow chart showing an example of the photographing process (S311) described above. In this process, still image shooting or moving image shooting (recording start or recording stop) is performed in accordance with a still image shooting instruction or a moving image shooting instruction by operating the imaging device 100 itself. When accepting a still image shooting instruction, the system control unit 50 performs still image shooting processing. Further, when receiving a moving image shooting instruction, the system control unit 50 performs recording start processing/recording stop processing depending on whether recording is being performed or not. Specifically, if the imaging apparatus 100 is already recording, recording stop processing is performed, and if recording is not being performed, recording start processing is performed. It should be noted that the imaging apparatus 100 performs still image shooting processing in the same manner as described above even when receiving a still image shooting instruction during recording. Details of the processing in the photographing processing (S311) will be described below.

The system control unit 50 determines whether or not there is an OFF camera unit (S3111). Then, if there is an OFF camera unit (S3111-YES), the system control unit 50 switches the OFF camera unit to ON (S3112).

Next, the system control unit 50 determines whether the instruction content of the imaging process is a still image imaging instruction or a moving image imaging instruction (S3113). In the case of a still image shooting instruction, the system control unit 50 executes still image shooting (S3115). In the case of a moving image shooting instruction, the system control unit 50 determines whether or not recording is in progress (S3114). If recording is not in progress (S3114-NO), the system control unit 50 starts recording (S3116). If recording is in progress (S3114-YES), the system control unit 50 stops recording (S3117). Note that data acquired by the imaging process is recorded in the memory 32 .

<Contents of Processing of Display Control Device 200>
4A to 4G are flowcharts showing an example of display control processing in the display control device 200. FIG. This processing is realized by loading the program recorded in the nonvolatile memory 203 into the memory 202 and executing it by the CPU 201 . When the power of the display control device 200 is turned on and the communication I/F or the external I/F becomes valid, the processing shown in FIG. 4A is started.

First, the CPU 201 determines whether the display control device 200 is communicating with the imaging device 100 (S401). When detecting communication by data reception of the communication I/F 210 (S401-YES), the CPU 201 executes the following processing. First, the CPU 201 receives VR image data (VR image) from the communication I/F 210 (S402). The CPU 201 also uses the image processing unit 204 to cut out a partial range of the VR image (composite image) from the received VR image data, and displays the cut out image on the display 205 (S403). Next, the CPU 201 receives the above imaging device status transmitted from the imaging device 100 (S404) and stores it in the memory 32. FIG.

Next, the CPU 201 detects whether or not there is an instruction to operate the display control device 200 . Specifically, the CPU 201 detects a button operation, a touch operation, or the like on the operation unit 206 or the touch panel 206a as an operation instruction. A case where the CPU 201 detects the presence/absence of a video display end operation instruction (S405), the presence/absence of a photographing operation instruction (S407), and the presence/absence of a display range change instruction (S409) will be described below.

First, when an operation instruction to end the video display is detected (S405-YES), the CPU 201 performs live video display end processing in the display control device 200 (S406). Next, when a photographing operation instruction is detected (S407-YES), the CPU 201 performs photographing instruction processing for transmitting a command corresponding to the operation instruction to the imaging apparatus 100 (S408). In this embodiment, the CPU 201 transmits to the imaging device 100 a command including either a still image shooting instruction or a moving image shooting start/stop instruction. Further, when the display control device 200 detects an instruction to change the display range (S409-YES), the CPU 201 causes the operation image processing unit 204 to change the cutout image range (S410). Details of the processing of steps S406, S408, and S410 will be described later.

In step S411, when no button operation or touch operation is performed (in the case of no operation), the CPU 201 uses the system timer 214 to start counting up the timer. Then, the CPU 201 determines whether or not the non-operation time has continued for a predetermined threshold value T or longer (S412). Note that the threshold value T can be, for example, 1 minute, but is not particularly limited. Also, as will be described later, the threshold value T may be changeable. If the non-operation time is less than the threshold T (S412-NO), the CPU 201 executes the process of step S402 again. If the non-operation time is equal to or longer than the threshold T (predetermined time or longer) (S412-YES), the CPU 201 performs power saving processing (S413). In the present embodiment, as power saving processing, among the plurality of images forming the VR image, the camera units other than the camera unit that captures the image within the display range are switched to the OFF state. Alternatively, when the image capturing apparatus 100 is recording, the images output by the camera units other than the camera unit that captures the image within the display range are converted into live video VR images without changing the ON state of the plurality of camera units. By not synthesizing, image transmission from the imaging device 100 to the display control device 200 is turned off for the camera unit that captures an image outside the display range. Details of the power saving process will be described later.

<<Processing for Ending Image Display (S406)>>
FIG. 4B is a flow chart showing an example of live video display end processing (S406) according to the present embodiment. Specifically, the CPU 201 clears the timer count in the memory 202 to zero (S4061), stops the display on the display 205 (S4062),
By stopping video reception and discontinuing communication with the imaging device 100 (S4063), end processing of live video display is performed. Then, as described above, in the imaging device 100, if "the live video is being transmitted to the outside and is not communicating with the display control device 200" (S308-YES), the system control unit 50 controls the display of the live video. The transmission to the device 200 is stopped (S309).

<<Shooting instruction processing (S408)>>
FIG. 4C is a flow chart showing an example of the photographing instruction process (S408) in this embodiment. In this embodiment, an example will be described in which the display control device 200 transmits a still image shooting command or a moving image shooting start/stop command to the imaging device 100 in response to a still image shooting instruction or a moving image shooting instruction. In addition, in the present embodiment, the display control device 200 instructs the imaging device 100 to start moving image shooting if the imaging device 100 is recording and to stop moving image shooting if the imaging device 100 is not recording in response to a moving image shooting instruction. do. Note that the display control device 200 may issue a moving image shooting start/stop command regardless of whether recording is in progress. Details of the photographing instruction process according to the present embodiment will be described below.

First, the CPU 201 clears the timer count in the memory 202 to zero (S4081). Then, the CPU 201 determines whether or not the content of the shooting instruction is a moving image shooting instruction (S4082). If it is a still image shooting instruction, the CPU 201 transmits a still image shooting instruction to the imaging device 100 (S4083). If it is a moving image shooting instruction, it is determined whether or not recording is in progress (S4084). If recording is not in progress (S4084-NO), the CPU 201 transmits a moving image shooting start command to the imaging device 100 (S4085). If recording is in progress (S4084-YES), the CPU 201 transmits a moving image shooting stop command to the imaging device 100 (S4086). After the processing of steps S4083, S4085, and S4086, the processing of step S402 shown in FIG. 4A is executed again.

<<Display Range Change Processing (S410)>>
FIG. 4D is a flowchart illustrating an example of display range change processing according to the present embodiment. In this process, when the display range is changed by operating the display control device 200 while the transmission image of the imaging device 100 is being displayed, the display control device 200 turns the camera unit of the imaging device 100 on as necessary. Send an ON command. Specifically, when the changed display range includes the imaging range of the OFF camera units of the imaging device 100, the display control device 200 turns ON the OFF camera units and starts image transmission. to the imaging device 100. Furthermore, when the display range after the change does not include the imaging range of the ON camera units among the camera units of the imaging device 100, the display control device 200 switches the ON camera units to OFF to stop image transmission. The command may be sent to the imaging device 100 . Here, when the camera units are recording, all the camera units are ON, so the display control device 200 composites the image output from the camera units that include the imaging range into the display range after the change. A command to start transmitting the VR image is transmitted to the imaging device 100 . Furthermore, the display control device 200 does not synthesize the image output from the camera unit that does not include the imaging range in the display range after the change with the VR image, thereby allowing the image output from the camera unit that does not include the imaging range. may be transmitted to the imaging device 100 to stop the transmission of . Details of the processing for changing the display range according to the present embodiment will be described below.

First, the CPU 201 causes the operation image processing unit 204 to change the display range of the VR image (composite image) transmitted from the imaging device 100 in step S303. Then, the CPU 201 cuts out a portion corresponding to the display range according to the operation from the VR image, and displays it as a live view on the display 205 (S4101). The CPU 201 then clears the timer count in the memory 202 to zero (S4102). Furthermore, the CPU 201 determines whether recording is in progress (S4103).

In step S4103, if recording is not in progress (S4103-NO), the CPU 201 determines whether there is an OFF camera unit (S4104). When the VR image received from the imaging device 100 has an end, the CPU 201 determines that there is an OFF camera unit. For example, when the camera unit b is turned off in the imaging device 100 , the imaging device 100 transmits to the display control device 200 a VR image generated only from the image 221 output from the camera unit a. As shown in FIG. 2D, the VR image generated only with the image 221 has the end 225 and the end 226, so the CPU 201 determines that there is an OFF camera unit. If there is an OFF camera unit (S4104-YES), the CPU 201 determines whether the display range has been moved so as to include the imaging range of the OFF camera unit (S4105). As shown in FIG. 2D, when the display range 224 moves to the display range 227 and the display range 227 is no longer filled with the VR image received from the imaging device 100, the CPU 201 detects the image of the camera unit whose display range is OFF. It is judged that it has moved to the range. When the display range is moved to be included in the imaging range of the OFF camera unit (S4105-YES), the CPU 201 transmits a command to turn ON the OFF camera unit to the imaging apparatus 100 (S4106). In this embodiment, there are only two cameras, the camera unit a and the camera unit b. Therefore, if the camera unit a is ON and the output image is being transmitted to the display control device 200, the camera unit a is OFF. is uniquely determined as the camera unit b. The above instructions are issued via the external I/F 209 or the communication I/F 210, for example. Although the display range is included in the imaging range of the ON camera unit, the CPU 201 moves to the vicinity of the edge of the imaging range of the ON camera unit. If it is expected that there is a high possibility of being included in the imaging range of the camera unit, a command to turn on the camera unit that has been turned off may be transmitted to the imaging device 100 . Alternatively, for example, when the distance between the coordinates of the center position or the overall position of the display range and the end of the image of the ON camera unit becomes equal to or less than a predetermined value, the imaging device issues a command to turn ON the OFF camera unit. 100. Further, in S4106, the CPU 201 may transmit to the imaging apparatus 100 a command to turn off the camera unit that captures an imaging range that does not include the display range at all after the display range has moved.

In step S4103, if recording is in progress (S4103-YES), the CPU 201 determines whether or not there is a camera unit for which image transmission is stopped (S4107). When the VR image received from the imaging device 100 has an end, the CPU 201 determines that there is a camera unit whose transmission is stopped. For example, when the camera unit b in the image capturing apparatus 100 is in suspension of transmission, the image capturing apparatus 100 transmits a VR image generated only from the image 221 output from the camera unit a to the display control device 200 as a live image. As shown in FIG. 2D, the VR image generated only with the image 221 has the end 225 and the end 226, so the CPU 201 determines that there is a camera unit whose transmission is stopped. If there is a camera unit whose transmission is stopped (S1407-YES), the CPU 201 determines whether the display range has been moved so as to include the imaging range of the camera unit whose image transmission is stopped (S4108). As shown in FIG. 2D, when the display range 224 moves to the display range 227 and the display range 227 is no longer filled with the VR images received from the imaging device 100, the CPU 201 changes the display range to the camera unit whose transmission is stopped. is determined to have moved to the imaging range of . When the display range is moved to be included in the imaging range of the camera unit whose transmission is stopped (S1408-YES), the CPU 201 captures a command to start transmitting the image output from the camera unit whose transmission is stopped. It is transmitted to the device 100 (S4109). Although the display range is included in the imaging range of the camera unit during image transmission, the CPU 201 moves to the vicinity of the end of the imaging range of the camera unit during image transmission, and if the movement continues as it is, the display When it is expected that the range is likely to be included in the imaging range of the camera unit whose image transmission is stopped, a command to start image transmission of the camera unit whose image transmission is stopped is transmitted to the imaging device 100. good too. Further, in step S4109, the CPU 201 may transmit to the imaging apparatus 100 a command to turn off image transmission of the camera unit that captures an imaging range that does not include the display range after the display range has moved. The above instructions are issued via the external I/F 209 or the communication I/F 210, for example. After this process, the process of step S402 shown in FIG. 4A is performed again.

<<Power Saving Processing (S413)>>
FIG. 4E is a flowchart showing an example of power saving processing (S413) according to this embodiment. In this processing, the CPU 201 performs power saving processing when the non-operation time is equal to or greater than the threshold value T. FIG. Specifically, the display control device 200 controls the imaging device 100 to capture an image of a range not displayed on the display 205 when a range of a part of the VR image (composite image) is displayed on the display 205. Send a command to turn off the camera unit. Further, when recording is in progress, the display control device 200 does not turn off the camera unit that captures the imaging range that does not include the display range, and transmits an instruction to turn off only image transmission. Here, when images captured by a plurality of camera units are included in the display range, the display control device 200 determines the camera units to be turned off according to the display range. Details of the power saving process according to the present embodiment will be described below.

First, the CPU 201 determines whether recording is in progress (S4131).

At step S4131, if recording is not in progress (S4131-NO), the CPU 201 determines the process to be executed according to the display range of the VR image being displayed on the display 205 (S4132). The CPU 201 compares the coordinate information indicating the imaging range of each camera unit and the coordinates of the display range received together with the VR image. Then, when only the image captured by the camera unit a is displayed on the display 205, the CPU 201 transmits a command to turn off the camera unit b as a camera unit other than the camera unit a to the imaging apparatus 100 (S4133). . When only the image captured by the camera unit b is displayed on the display 205, the CPU 201 transmits a command to turn off the camera unit a as a camera unit other than the camera unit b to the imaging apparatus 100 (S4134). When the images (boundaries) of both the camera units a and b are displayed on the display 205, the CPU 201 performs boundary processing (S4135). Details of the boundary processing will be described later.

At step S4131, if recording is in progress (S4131-YES), the CPU 201 determines the process to be executed according to the display range of the VR image being displayed on the display 205 (S4136). The CPU 201 compares the coordinate information indicating the imaging range of each camera unit and the coordinates of the display range received together with the VR image. Then, when only the image captured by the camera unit a is displayed on the display 205, the CPU 201 stops transmitting the live video synthesized with the image output from the camera unit b as the camera unit other than the camera unit a. A command is transmitted to the imaging device 100 (S4137). When only the image captured by the camera unit b is displayed on the display 205, the CPU 201 issues an instruction to stop transmission of the live video synthesized with the image output from the camera unit a as a camera unit other than the camera unit b. It is transmitted to the imaging device (S4138). If the display 205 includes images from both the camera units a and b, the CPU 201 performs boundary processing during recording (S4139). The details of the border processing during recording will be described later. After the processing of steps S4133 to S4135 and S4137 to S4139, step S402 is executed again.

<<Boundary processing (S4135)>>
FIG. 4F is a flow chart showing an example of the boundary processing (S4135) described above. When the boundary processing is executed, the CPU 201 determines whether or not the display size of the image captured by the camera unit a in the display range of the VR image is equal to or larger than the display size of the image captured by the camera unit b (S41351). ).

When the image of the camera unit b occupies a larger area than the image of the camera unit a in the display range (S41351-NO), the display 205 is displayed so that it is included only in the image captured by the camera unit b in the live video. is changed (S41352), and the camera part a is turned off
A command to change to F is transmitted to the imaging device (S41353). In addition, when the image of camera unit a occupies a larger area than the image of camera unit b in the display range (S41351-YES), the display is made so that only the image captured by camera unit a in the live video is included. 205 is changed (S41354), and a command to turn off the camera unit b is transmitted to the imaging device (S41355).

<<Boundary processing during recording (S4139)>>
FIG. 4G is a flow chart showing an example of the boundary process during recording (S4139) described above. When the boundary process during recording is executed, the CPU 201 determines whether or not the display size of the image captured by the camera unit a within the display range of the VR image is equal to or larger than the display size of the image captured by the camera unit b. (S41391).

When the image of the camera unit b occupies a larger area than the image of the camera unit a in the display range (S41391-NO), the CPU 201 causes the live image to be included only in the image captured by the camera unit b. The display range of the display 205 is changed (S41392). Then, the CPU 201 transmits a command to stop the image transmission of the camera unit a to the imaging device 100 (S41393). In addition, when the image of camera unit a occupies a larger area than the image of camera unit b in the display range (S41391-YES), the CPU 201 determines that only the image captured by camera unit a in the live video contains the image captured by camera unit a. The display range of the display 205 is changed (S41394) as follows. Then, the CPU 201 transmits a command to stop the image transmission of the camera section b to the imaging device 100 (S41395).

After the processing of steps S41393 and S41395, a command to resume image transmission of the camera units a and b may be transmitted to the imaging device under a predetermined condition. The predetermined condition is, for example, when the user input instruction to put the images captured by the camera units a and b into the display range continues for a predetermined period or longer, or when the instruction input is repeated twice or more within the predetermined period. and the like.

<Startup time reduction processing>
In the present embodiment, in order to shorten the delay time accompanying activation of the camera unit, the stop target function of the imaging device 100 is changed according to the position of the display range. Specifically, depending on where the display range is located in the VR image, power supply to the imaging unit 22 of the imaging device 100 is stopped, power supply to the A/D converter 23 is stopped, image processing and display control device Either stopping the image transmission process to 200 is performed. In this embodiment, an example in which two rectangular ranges are provided as the predetermined range will be described, but the number and shape of the ranges are not particularly limited. Whether or not it is within the area is determined based on the coordinates of the cut-out display range, such as the center position of the cut-out display range or the overall position of the display range. Also, although an example in which the startup time shortening process is performed in the display control device 200 will be described, this process may be performed in the imaging device 100 .

FIG. 5A shows a state in which a virtual VR image obtained by combining a captured image 504 captured by the camera unit a and a captured image 505 captured by the camera unit b is developed on a plane in a pseudo manner. A region 501 indicates a clipped display range displayed on the display 205 of the display control device 200 . A range 502 and a range 503 indicate the predetermined ranges described above. In the example shown in FIG. 5A, a captured image 504 captured by the camera unit a is transmitted from the imaging device 100 to the display control device 200, and a part thereof is displayed on the display 205 as a display range. An example of changing the state of the camera unit b according to the display range when a predetermined time (threshold value T) has elapsed will be described below.

FIG. 5B is a flowchart illustrating an example of boot time reduction processing. When the entire display range is included inside the range 503 (S501-YES), the CPU 201 stops power supply to the camera imaging unit 22b and the A/D converter 23b of the imaging device 100 (S502).

If the area outside the range 503 is included in the display range and the entire display range is included inside the range 502 (S501-NO and S503-YES), the CPU 201 controls the A/D converter of the imaging device 100. 23b is stopped (S504). Power is supplied to the camera imaging section 22b.

If the area outside the range 502 is included in the display range (S501-NO and S503-NO), the CPU 201 supplies power to the camera imaging unit 22b and the A/D converter 23b, but the A/D converter No image processing is performed on the image b output from 23b. Then, the image transmission is stopped by not synthesizing the image processing result of the image b with the VR image (S505).

<Change processing of non-operation time>
In the present embodiment, the threshold value T of the non-operation time described above is changed by the user performing a setting operation. Note that although an example of changing using a GUI displayed on the display of the display control device 200 will be described below, processing for changing the non-operation time may be performed by operating the imaging device 100 .

FIG. 6A shows an example of a screen for changing the no-operation time setting displayed on the display 205 of the display control device 200. FIG. In addition, the "power saving state" in FIG. 6A includes turning off the camera unit or stopping image transmission of the camera unit.

FIG. 6B is a flowchart illustrating an example of non-operation time change processing. When the no-operation time change instruction is executed, the CPU 201 opens the setting menu of the display control device 200 (S601). When the non-operation time setting is changed, the CPU 201 clears the timer counter (S603) and records the selected or input time in the memory 102 (S604). The non-operation time setting can correspond to either selection of a fixed value or input of an arbitrary time. Further, the CPU 201 closes the setting menu when there is an operation instruction to close the setting menu (S605). It is also possible to set no operation time by selecting "timer off" in FIG. 6A.

<Advantageous effects of the present embodiment>
As described above, the display control device according to the present embodiment can appropriately reduce the power consumption of the imaging device by appropriately controlling the driving of the camera unit in the imaging device by switching the display range. In addition, when the imaging device is recording a moving image, the use state of the imaging device can be changed by recording captured images obtained by a plurality of camera units and stopping transmission of captured images that are unnecessary for live view display. It is possible to suppress power consumption corresponding to Furthermore, by changing the stop state of the imaging device, it is possible to suppress the delay in the time required to start the camera while suppressing power consumption.

(Modification)
In the above-described embodiments, an imaging apparatus having two camera units was described as an example, but three or more camera units may be provided. In this case, the target for which the above-mentioned camera section is turned off may be a camera section other than the camera section that captures the image displayed on the display control device. Further, in the boundary processing described above, the cameras other than the camera that captures the image with the largest display range may be turned off. Note that the above functions may be provided in a control system including a plurality of imaging devices having one or more camera units and a display control device.

In the above-described embodiment, the image displayed on the display control device is not processed to reduce power consumption.

In the above-described embodiment, the process of saving power is described when no operation is received for a predetermined time or more, but the above process may be performed when a predetermined operation is received from the user. Examples of predetermined operations include user operations performed using buttons and GUIs of the imaging device or the display control device. Another example of the predetermined operation is an operation of changing the position or orientation of the display control device. For example, the camera unit may be turned off when the user faces down the display unit of the display control device, and may be turned on in other states.

In the above-described embodiment, an example in which the imaging device stops the imaging process in (part of) the imaging unit or stops the image transmission process according to the command of the display control device has been described, but based on the range information of the display control device The above processing may be performed according to the judgment of the imaging device. Specifically, the imaging device receives range information of a part of the VR image (composite image) displayed on the display control device from the display control device. Then, the imaging device performs imaging processing in the imaging unit that captures an image of a range that is not displayed on the display control device when a partial range of the VR image is displayed on the display control device based on the range information. You may stop or stop the transmission process of an image.

In the above-described embodiment, an example in which the imaging device and the display control device are separate devices has been described. may be provided.

(Second embodiment)
The operation of the imaging device 100 according to the second embodiment will be described with reference to the flowchart of FIG. It is assumed that the imaging device 100 has been supplied with power in advance, performed startup processing, established communication with the display control device 200 , and received a request for live view display from the display control device 200 .

First, the system control unit 50 controls all camera units (camera unit a and camera unit b in this embodiment) to be driven in the normal mode (S701). The system control unit 50 performs image processing on the images output from all the camera units, combines the image processing results, and generates a VR image (S702). The system control unit 50 transmits the VR image generated in step S702 as a live image to the display control device 200 (S703).

Then, the system control unit 50 receives, from the display control device 200, coordinate information (display range information) indicating the display range displayed on the display 205 in the VR image transmitted in step S703 (S704). The system control unit 50 compares the coordinate information indicating the combining positions of the images output from the respective camera units forming the VR image with the coordinate information of the display range received from the display control device 200 (S705).

As a result of the comparison, if there is a camera unit that outputs an image that is not included in the display range (Yes in S706), the system control unit 50 drives the camera unit that outputs an image that is not included in the display range in power saving mode. (S707). In the power saving mode, the system control unit 50 stops all the functions of the camera unit, or disables at least some of the functions of the camera unit (for example, at least one of the imaging unit 22 and the A/D converter 23). ), or lower the frame rate of the image output from the camera unit. In other words, in the power saving mode, the camera section is controlled so as to consume less power than when operating in the normal mode. Then, the system control unit 50 performs image processing on the images output from the camera units driven in the normal mode, and combines the image processing results to generate a VR image (S708). The system control unit 50 transmits the VR image generated in step S708 as a live image to the display control device 200 (S709).

Then, in S710 and S711, the system control unit executes the same processing as in S704 and S705. Based on the coordinate information indicating the display range, the system control unit determines whether the display range is near the end of the VR image (S712). When the VR image is generated only from images output from some camera units, the VR image 223 has ends 225 and 226 as shown in FIG. 2D. When the position of the display range 224 is within a predetermined distance from the end of the VR image, the system control unit 50 determines that the display range is near the end of the VR image. Then, the system control unit 50 specifies the camera unit that outputs the image of the imaging range adjacent to the end in the vicinity of the display range, and controls the specified camera unit to be driven in the normal mode (S713). In step S713, the system control unit 50 controls to switch the specified camera unit from the power saving mode to the normal mode and drive it. Then, the system control unit 50 performs image processing on the images output from the camera units that are being driven in the normal mode, including the identified camera units, and combines the image processing results to generate a VR image (S714). . The VR image generated in step S712 is transmitted to the display control device 200 as a live image (S715).

Then, the system control unit 50 determines whether or not a recording start instruction has been received (S716). The system control unit 50 accepts a recording start instruction when there is an input to a predetermined operation member provided in the imaging device 100, or when the system control unit 50 receives a recording start instruction from the display control device 200. I judge that. When the recording start instruction is received (Yes in S716), the system control unit 50 controls all the camera units to be driven in the normal mode (S717), performs image processing on the images output from all the camera units, The image processing results are combined to generate a VR image for recording (S718), the VR image is transmitted as a live image to the display control device 200 (S719), and the VR image is recorded on the recording medium 120 (S720).

Then, the system control unit 50 determines whether or not a recording stop instruction has been received (S721). The system control unit 50 accepts a recording stop instruction when there is an input to a predetermined operation member provided in the imaging device 100, or when the system control unit 50 receives a recording stop instruction from the display control device 200. I judge that. Upon receiving the recording stop instruction (Yes in S721), the system control unit 50 stops the recording process and determines whether or not an instruction to end the live view display has been received from the display control device 200 (S722). If an instruction to end the live view display is received (Yes in S722), this process ends. If the live view display end instruction is not received (No in S722), the process returns to step S704. If the recording stop instruction is not received (No in S721), the process returns to step S718 to continue the recording process.

The operation of the display control device 200 according to the second embodiment will be described with reference to the flowchart of FIG. It is assumed that the display control device 200 is supplied with power, performs startup processing, establishes communication with the imaging device 100 , and transmits a request for live view display to the imaging device 100 in advance.

First, the CPU 201 receives the VR image transmitted from the imaging device 100 in S703 (S801), and displays a part of the VR image as a display range on the display 205 (S802). In step S802, as the initial position, for example, the center of the VR image is displayed as the display range. Then, the CPU 201 determines whether an instruction to change the display range of the VR image has been received (S803). The CPU 201 determines that an instruction to change the display range has been received when there is an operation on the touch panel 206 or a change in the posture of the display control device 200 . When the CPU 201 determines that an instruction to change the display range has been received (Yes in S803).
s), the coordinate information indicating the changed display range is transmitted to the imaging device 100 (S804). The CPU 201 receives the VR image transmitted from the imaging device 100 in S709 or S715 (S805), and displays a part of the VR image corresponding to the changed display range on the display 205 (S806).

Then, when receiving the recording start instruction (Yes in S807), the CPU 201 transmits the recording start instruction to the imaging device 100 (S808). The CPU 201 receives the VR image transmitted from the imaging device 100 in S719 (S809), and displays a part of the VR image corresponding to the selected display range on the display 205 (S810). When the CPU 201 receives an instruction to change the display range of the VR image (Yes in S811), the CPU 201 displays a part of the VR image corresponding to the changed display range on the display 205 (S812). Note that the imaging apparatus 100 drives all the camera units in the normal mode during recording, so there is no need to switch between the normal mode and the power saving mode for each camera unit. Therefore, when the imaging device 100 is recording, the display control device 200 does not transmit the coordinate information of the display range to the imaging device 100 even if the display range of the VR image is changed.

When the CPU 201 receives the recording stop instruction (Yes in S813), the CPU 201 transmits the recording stop instruction to the imaging device 100 (S814). If the CPU 201 does not receive a recording stop instruction (No in S813), the process returns to step S809.

When the CPU 201 receives the live view display end instruction (Yes in S815), the CPU 201 transmits the live view display end instruction to the imaging device 100 (S816). If the CPU 201 does not receive an instruction to end the live view display (No in S815), the process returns to step S801.

According to the present embodiment, each camera unit of the imaging device 100 is switched between normal mode and power saving mode in accordance with a change in the display range of the VR image displayed on the display 205 of the display control device 200 . As a result, the power consumption of the imaging device 100 can be suppressed by driving the camera unit that captures an image of a range not displayed on the display 205 of the display control device 200 in the power saving mode.

(Third embodiment)
The operation of the imaging device 100 according to the third embodiment will be described with reference to the flowchart of FIG. It is assumed that the imaging device 100 has been supplied with power in advance, performed startup processing, established communication with the display control device 200 , and received a request for live view display from the display control device 200 .

First, the system control unit 50 controls all camera units (camera unit a and camera unit b in this embodiment) to be driven in the normal mode (S901). The system control unit 50 transmits images output from all camera units to the display control device 200 (S902).

Then, the system control unit 50 receives, from the display control device 200, information indicating an image being displayed or an image not being displayed on the display 205 among the images transmitted in step S902. Then, if the system control unit 50 determines that there is a hidden image (Yes in S903), it identifies the camera unit that has output the hidden image, and switches to drive in the power saving mode (S904). The system control unit 50 transmits images output from the camera units driven in the normal mode to the display control device 200, except for the camera units in the power saving mode (S905).

In addition, the system control unit 50 receives information indicating the position of each image constituting the live image displayed on the display 205 and the display range of the live image, and determines whether the display range is near the end of the live image. Determine (S906). If the display range is near the end of the live image (Yes in S906), the camera unit that outputs the image adjacent to the image including the end is identified and switched to drive in the normal mode (S907). Then, the image of the camera unit driven in the normal mode is transmitted to the display control device (S908).

Then, the system control unit 50 determines whether or not a recording start instruction has been received (S909). When the recording start instruction is accepted (Yes in S909), the system control unit 50 controls all the camera units to be driven in the normal mode (S910), performs image processing on the images output from all the camera units, The image processing results are synthesized to generate a VR image for recording (S911), and the VR image is recorded on the recording medium 120 (S912). The system control unit 50 transmits images output from all camera units to the display control device 200 (S913).

Then, the system control unit 50 receives, from the display control device 200, information indicating the image being displayed or not being displayed on the display 205 among the images transmitted in step S913. Then, if it is determined that there is a hidden image (Yes in S914), the system control unit 50 identifies the camera unit that output the hidden image, and controls the display of the image output from the identified camera unit. The transmission to the device 200 is stopped (S915).

Further, when the system control unit 50 receives information from the display control device 200 indicating that the display range of the display 205 is near the end of the live image (Yes in S916), the system control unit 50 outputs an image adjacent near the end. The camera unit to be used is specified, and the transmission of the image output from the specified camera unit to the display control device 200 is resumed (S917).

When the system control unit 50 receives the recording stop instruction (Yes in S918), the system control unit 50 stops the recording process and determines whether or not an instruction to end the live view display has been received from the display control device 200. (S919). If an instruction to end the live view display is received (Yes in S919), this process ends. If the live view display end instruction is not received (No in S919), the process returns to step S903. If the recording stop instruction is not accepted (No in S721), the process returns to step S911 to continue the recording process.

The operation of the display control device 200 according to the third embodiment will be described with reference to the flowchart of FIG. It is assumed that the display control device 200 is supplied with power, performs startup processing, establishes communication with the imaging device 100 , and transmits a request for live view display to the imaging device 100 in advance.

First, the CPU 201 receives an image output from each camera unit transmitted from the imaging apparatus 100 in S902 (S1001), and generates a VR image by synthesizing a plurality of received images (S1002). A part of the VR image is displayed on the display 205 as a display range (S1003). In step S1003, as the initial position, for example, the center of the VR image is displayed as the display range. When receiving an instruction to change the display range of the VR image (Yes in S1004), the CPU 201 displays a part corresponding to the changed display range on the display 205 (S1005).

Then, the CPU 201 generates information indicating whether the image is being displayed or not for each image forming the VR image, and transmits the information to the imaging device 100 (S1006). In response to this, the imaging apparatus 100 executes the processing from S903 to S905. The CPU 201 receives the image transmitted from the imaging device 100 in S905 (S1007), synthesizes the received images to generate a VR image, and displays a portion of the VR image corresponding to the current display range. 205 (S1008).

In addition, the CPU 201 generates information indicating the display range in the VR image generated as the live image and the positions of the images forming the VR image, and transmits the information to the imaging device 100 (S1009). In response to this, the imaging apparatus 100 executes the processing from S906 to S908. The CPU 201 receives the image transmitted from the imaging device 100 in S908 (S1010), synthesizes the received images to generate a VR image, and displays a portion of the VR image corresponding to the current display range. 205 (S1011).

Then, when receiving the recording start instruction (Yes in S1012), the CPU 201 transmits the recording start instruction to the imaging device 100 (S1013). The CPU 201 receives the image of each camera unit transmitted from the imaging device 100 in S913 (S1014), and generates a VR image by synthesizing a plurality of received images (S1015). A part of the VR image is displayed on the display 205 as a display range (S1016). When the CPU 201 receives an instruction to change the display range of the VR image (Yes in S1017), the CPU 201 displays a part of the VR image corresponding to the changed display range on the display 205 (S1018).

Then, the CPU 201 generates information indicating whether it is being displayed or not for each image forming the VR image, and transmits the information to the imaging device 100 (S1019). In response to this, the imaging device 100 executes the processes of S914 and S915. The CPU 201 receives the image transmitted from the imaging device 100 in S915 (S1020), synthesizes the received images to generate a VR image, and displays a portion of the VR image corresponding to the current display range. 205 (S1021).

Further, the CPU 201 generates information indicating the display range in the VR image generated as the live image and the positions of the images forming the VR image, and transmits the information to the imaging device 100 (S1022). In response to this, the imaging device 100 executes the processes of S916 and S917. The CPU 201 receives the image transmitted from the imaging device 100 in S917 (S1023), synthesizes the received images to generate a VR image, and displays a portion of the VR image corresponding to the current display range. 205 (S1024).

When the CPU 201 receives the recording stop instruction (Yes in S1025), the CPU 201 transmits the recording stop instruction to the imaging device 100 (S1026). If the CPU 201 does not receive a recording stop instruction (No in S1025), the process returns to step S1019.

When the CPU 201 receives the live view display end instruction (Yes in S1027), the CPU 201 transmits the live view display end instruction to the imaging device 100 (S1028). If the CPU 201 does not receive an instruction to end the live view display (No in S1027), the process returns to step S1006.

According to the present embodiment, each camera unit of the imaging device 100 is switched between normal mode and power saving mode in accordance with a change in the display range of the VR image displayed on the display 205 of the display control device 200 . As a result, the power consumption of the imaging device 100 can be suppressed by driving the camera unit that captures an image of a range not displayed on the display 205 of the display control device 200 in the power saving mode.

Further, according to the present embodiment, even when the imaging apparatus 100 is performing recording processing and all the camera units need to be driven in the normal mode, the display range of the VR image displayed on the display 205 of the display control apparatus 200 is reduced. Depending on the change, whether or not to transmit an image is switched for each camera unit of the imaging device 100 . As a result, the power consumption of the imaging device 100 can be suppressed by stopping the transmission of the image of the camera unit that captures the image of the range not displayed on the display 205 of the display control device 200 .

(others)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

100: Imaging Device a: Camera Section b: Camera Section 22a, 22b: Imaging Section 24: Image Processing Section 50: System Control Section 200: Display Control Device 204: Image Processing Section 205: Display

Claims (7)

a plurality of imaging processing means for outputting images obtained by imaging subjects in different directions;
generating means for generating a synthesized image by synthesizing the images output by the plurality of imaging processing means;
a recording means for recording the composite image on a recording medium;
Acquisition means for acquiring display range information indicating a display range of the composite image to be displayed on a screen;
controlling the plurality of imaging processing means to be driven in a first mode while the composite image is being recorded on the recording medium, regardless of the display range information;
While the synthesized image is not recorded on the recording medium, the imaging processing means for outputting an image corresponding to the displayed range of the synthesized image is driven in the first mode based on the display range information. and a control means for controlling the imaging processing means for outputting an image corresponding to a non-display range of the composite image so as to be driven in a second mode that saves power compared to the first mode;
An imaging device characterized by comprising: further comprising transmission means for transmitting the composite image to an external device;
The acquisition means acquires from the external device the display range information indicating the display range displayed on the screen of the external device.
2. The imaging device according to claim 1, wherein: further comprising transmission means for transmitting the plurality of images output by the plurality of imaging processing means to an external device;
The acquisition means acquires, from the external device, display range information indicating a display range to be displayed on a screen of the external device, in a composite image generated by synthesizing the plurality of images in the external device.
2. The imaging device according to claim 1, wherein: Based on the display range information, the transmission means transmits to the external device an image output by the imaging processing means corresponding to a displayed range of the synthesized image, and a non-display range of the synthesized image. and not transmitting the image output by the imaging processing means corresponding to the external device,
4. The imaging device according to claim 3, characterized in that: The imaging processing means includes an imaging section and an A/D converter that converts an analog signal output from the imaging section into a digital signal, and the control means controls at least one of the imaging section and the A/D converter. driving the imaging processing means in the second mode by deactivating one;
5. The imaging apparatus according to any one of claims 1 to 4, characterized by: A control method for an imaging device having a plurality of imaging processing means for outputting images obtained by imaging subjects in different directions,
a generating step of generating a synthesized image by synthesizing the images output by the plurality of imaging processing means;
a recording step of recording the composite image on a recording medium;
an obtaining step of obtaining display range information indicating a display range of the composite image displayed on a screen;
controlling the plurality of imaging processing means to be driven in a first mode while the composite image is being recorded on the recording medium, regardless of the display range information;
While the synthesized image is not recorded on the recording medium, the imaging processing means for outputting an image corresponding to the displayed range of the synthesized image is driven in the first mode based on the display range information. and a control step of controlling the imaging processing means for outputting an image corresponding to a non-display range of the composite image so as to be driven in a second mode that saves power compared to the first mode;
A control method for an imaging device, comprising: A program for causing a computer to function as each means of the imaging apparatus according to any one of claims 1 to 5.

Images