JP2020088559A - Electronic device and program - Google Patents

Abstract

To solve the problem in which, with an imaging apparatus capable of omnidirectional photographing, since there is no place to avoid being photographed from a time a photographer gives an instruction to start photographing until the photographer moves to a position away from the imaging apparatus, the photographer may be photographed.SOLUTION: The electronic device controls not to record a video during a time from a photographer gives an instruction to start photographing until the photographer moves to a position away from an imaging apparatus or not to reproduce the video.SELECTED DRAWING: Figure 3

Description

The present invention relates to an electronic device and a program capable of capturing or reproducing a VR image.

2. Description of the Related Art Conventionally, there is an imaging device that is capable of shooting a landscape in all directions around by including a plurality of wide-angle lenses. In addition, there is an imaging device or a playback device capable of playing back VR (Virtual Reality) content created based on images taken by these imaging devices (see Patent Document 1).

JP, 2014-222825, A

In an imaging device capable of omnidirectional shooting, for example, when a tripod is attached to the imaging device and an attempt is made to shoot the omnidirectional surroundings, a photographer operates the imaging device to instruct shooting start, and then the imaging device is operated. There is no place to avoid taking pictures before moving to a distant position. Therefore, unlike the intention of the photographer, there is a problem in that the photographer may be photographed in close-up or the image thus photographed may be reproduced.

One of the inventions according to the present application is to detect whether a tripod is joined, and whether or not the size of the face image of the photographer recognized from the VR image captured by the image capturing unit is larger than a predetermined size. If the determination means and the tripod are joined and the size of the face image of the photographer is larger than a predetermined size, the VR image captured by the image capturing means after a predetermined time has elapsed after receiving the shooting instruction. It is characterized in that it is provided with a generating means for compressing and encoding the above to generate VR moving image data, and a recording means for recording the VR moving image data in a recording medium.

According to the present invention, in the case of shooting with an imaging device capable of omnidirectional shooting, the photographer is taken a close-up image from the time when the shooting start is instructed to the time when the user moves to a position away from the imaging device. However, the video during that time is not shot or played. Therefore, it is possible to obtain an effect that an image suitable for the photographer's intention can be provided.

External view showing an example of the camera 100 Block diagram showing an example of the hardware configuration of the camera 100 The flowchart figure which shows an example of the operation|movement which the camera 100 performs. The flowchart figure which shows an example of the operation|movement which the camera 100 performs. The flowchart figure which shows an example of operation which the camera 100 performs.

Hereinafter, preferred embodiments of the present invention will be illustratively described in detail with reference to the drawings. However, the constituent elements described in this embodiment are merely examples, and the scope of the present invention is not intended to be limited thereto.

(First embodiment)
First, FIGS. 1A and 1B show the appearance of a camera 100 as an example of the electronic device of the present embodiment. The camera 100 is a lens-integrated digital camera. The camera 100 may be a single-lens reflex camera, a mirrorless single-lens camera, or the like. The present invention can be applied to various image pickup devices to which a wide-angle lens is attached.

FIG. 1A shows the appearance of the front surface of the camera 100. The display unit 141 displays a captured image and various information. The image displayed on the display unit 141 can be switched according to a shooting preparation instruction or a reproduction instruction. The touch panel 13 provided on the display surface of the display unit 141 receives a touch operation by the photographer. The release button 11 is a switch that can be pressed halfway and fully. When the release button 11 is half-pressed, the camera 100 is instructed to perform shooting preparation, and when the release button 11 is fully pressed, the camera 100 is instructed to shoot. .. The lens 111 is a wide-angle lens, and forms a light flux incident on the lens 111 on an image sensor 212 (not shown in FIG. 1A).

FIG. 1B shows the external appearance of the side surface of the camera 100. The lens 101 and the lens 111 are arranged in a pair in front of and behind the camera 100, and the angle of view of each of the lens 101 and the lens 111 is 280 degrees. That is, the angle of view of 360 degrees is satisfied by the lenses 101 and 111. Therefore, an image corresponding to an angle of view of 280 degrees is captured by each of the image pickup elements 212 of the image pickup element 202. Then, the two images picked up by the two image pickup devices 202 and 212 form a VR image corresponding to the angle of view of 360 degrees. Then, the tripod 120 is joined to the joining portion provided on the bottom surface of the camera 100.

A VR image is captured by the image capturing elements 202 and 212. The VR image is an image that can be displayed in VR. The VR image is an omnidirectional image (omnidirectional image) captured by an omnidirectional camera (omnidirectional camera), or a panoramic image having a wider image range (effective image range) than the display range that can be displayed on the display unit at one time. Etc. shall be included. The VR image includes not only still images but also moving images and live view images (images obtained from the camera in almost real time). The VR image has an image range (effective image range) of 360 degrees in the vertical direction (vertical angle, angle from zenith, elevation angle, depression angle, altitude angle) and 360 degrees in the horizontal direction (horizontal angle, azimuth angle) at the maximum. To have. Further, even if the VR image is less than 360 degrees in the vertical direction and less than 360 degrees in the horizontal direction, the VR image can be displayed in a wide field angle (viewing range) wider than the field angle that can be captured by a normal camera, or can be displayed at once on the display means. Images with a wider video range (effective video range) than the display range are also included. For example, an image taken by a omnidirectional camera capable of shooting a subject in a field of view (angle of view) of 360 degrees in the left-right direction (horizontal angle, azimuth angle) and a vertical angle of 210 degrees centered on the zenith (zenith) is It is a type of VR image. Further, for example, an image captured by a camera capable of capturing a subject having a field of view (view angle) of 280 degrees in the left-right direction (horizontal angle, azimuth angle) and a vertical angle of 280 degrees centered on the horizontal direction is a VR image. It is a kind. That is, an image that has a visual range of 260 degrees (±80 degrees) or more in each of the vertical direction and the horizontal direction and that has a wider visual range than a human can visually recognize at once is a VR image. Is a kind of. When this VR image is VR-displayed, by changing the attitude of the display unit in the left-right rotation direction, it is possible to view an omnidirectional video image that is seamless in the left-right direction (horizontal rotation direction). In the up-down direction (vertical rotation direction), you can watch a seamless omnidirectional image within a range of ±205 degrees when viewed from directly above (zenith), but the image is visible in a range that exceeds 205 degrees from directly above. It is a blank area that does not exist. The VR image can also be said to be an “image in which the video range is at least a part of the virtual space (VR space)”.

The VR display is a display method capable of changing the display range, in which an image in a visual field range corresponding to the attitude of the display means is displayed in the VR image. When a head mounted display (HMD), which is a display unit, is attached and viewed, an image in a visual field range corresponding to the direction of the user's face is displayed. For example, in a VR image, an image with a viewing angle (angle of view) centered at 0° in a horizontal direction (a specific azimuth, for example, north) and 90° in a vertical direction (90° from the zenith, that is, horizontal) at a certain point in time. Is displayed. From this state, if the posture of the display means is reversed upside down (for example, if the display surface is changed from southward to northward), 280 degrees (reverse azimuth, for example, south) of the same VR image is displayed vertically. The display range is changed to an image with a viewing angle centered at 90 degrees (horizontal) in the direction. In the case where the user is watching the HMD, if the user turns his face from north to south (that is, if he faces back), the image displayed on the HMD will change from the image in the north to the image in the south. is there. With such a VR display, the user can be visually provided with a feeling as if he/she were in the VR image (in the VR space). A smartphone equipped with VR goggles (head mount adapter) can be said to be a type of HMD. The display range may be moved (scrolled) according to a user operation on the touch panel, direction buttons, or the like, instead of the change in posture.

The display unit 141 may be a device separate from the camera 100. For example, the display unit 141 may be provided on a smartphone or the like that is separate from the camera 100. In this case, the electronic device that captures and records the VR image and the electronic device that reproduces the VR image are realized by separate devices.

Next, an example of the hardware configuration of the camera 100 will be described with reference to FIG. In FIG. 2, each function is shown as a separate component, but each function may be implemented by hardware such as one or more ASICs or programmable logic arrays (PLA). Each function may be realized by executing software by a programmable processor such as a CPU or MPU.

The lens 101 is a single focus lens including a diaphragm mechanism, and the focus position of the lens 101 is pan focus.

The image sensor 202 has a configuration in which a plurality of pixels including photoelectric conversion elements are two-dimensionally arranged. The image pickup element 202 photoelectrically converts, that is, picks up an object image (optical image) formed by a photographing optical system including the lens 101 for each pixel, generates an analog signal, and a digital signal is generated by an A/D conversion circuit. And outputs RAW image data in pixel units.

The memory I/F unit 206 writes the RAW image data for all pixels output from the image sensor 202 into the memory 207. The memory I/F unit 206 also reads the RAW image data stored in the memory 207 and outputs it to the image processing unit 208. The memory 207 is a volatile recording medium that stores all pixel RAW image data for several frames.

The image processing unit 208 performs processing such as gamma correction, color separation, and color difference matrix on the RAW image data for all pixels from the memory I/F unit 216, adds a synchronization signal, and records or displays an image. Output as data. The image processing unit 208 also obtains a photometric value from the brightness value of each pixel of the RAW image data input to the image processing unit 208, and provides the photometric value to the camera control unit 220 via the bus 121.

The lens 111, the image sensor 212, the memory I/F unit 216, the memory 217, and the image processing unit 218 are the same as the lens 101, the image sensor 202, the memory I/F unit 206, the memory 207, and the image processing unit 208. It has the function of. The lens 111, the image pickup device 212, the memory I/F unit 216, the memory 217, and the image processing unit 218 are the same as the lens 101, the image pickup device 202, the memory I/F unit 206, the memory 207, and the image processing unit 208. Is processed.

The camera control unit 220 includes a CPU and controls the overall operation of the camera 100. The RAM 260 is used as a work area (work memory) of the camera control unit 220. In the RAM 260, constants and variables for the operation of the camera control unit 220 are recorded, and programs and the like are expanded. The ROM 280 stores a computer program and the like for operating the camera control unit 220. For example, the camera control unit 220 controls the entire operation of the camera 100 by expanding the computer program stored in the ROM 280 into the RAM 260 and executing the computer program.

The gain control unit 203, the shutter control unit 204, and the aperture control unit 205 are used for exposure control. The camera control unit 220 controls the control units 203 to 205 based on the photometric value provided from the image processing unit 208 and the image processing unit 218 or the operation parameter manually set by the user. The gain controller 203 controls the gains of the image sensor 202 and the image sensor 212. The shutter control unit 204 controls the shutter speed of the image sensor 202 and the image sensor 212. The diaphragm control unit 205 controls the diaphragm mechanism of the lens 101 and the lens mechanism of the lens 111.

The recording medium I/F unit 230 is an interface for connecting the recording medium 231 to the camera 100. The recording medium I/F unit 230 records the video data input from the image processing unit 208, the image processing unit 218, or the GPU 242 in the recording medium 231, and reads the recorded video data from the recording medium 231. The recording medium 231 is composed of a semiconductor memory or the like.

The display I/F unit 240 outputs to the display unit 141 the video data from the image processing unit 208 and the image processing unit 218 and the image data drawn in the VRAM (Video RAM) 243 by the GPU 242. The image displayed on the display unit 141 is mainly used by the photographer for confirming the captured image.

Based on an instruction from the camera control unit 220, the GPU 242 performs image conversion processing, image cutout processing, enlargement processing, distortion correction, and the like on the video data output from the image processing unit 208 and the image processing unit 218, and rendering to the VRAM 243. It is an engine. The GPU 242 also performs a process of displaying various information of the camera 100 and a menu screen on the VRAM 243 in a superimposed manner. The touch panel button 14 described above is drawn by the GPU 242 as a GUI (Graphical User Interface) and is displayed on the display unit 141 via the display I/F unit 240.

The communication I/F unit 250 connects to a smartphone and a PC (not shown) outside the camera 100 via the communication unit 251, and transfers the image data displayed on the display unit 141 to them. The communication I/F unit 250 also receives a control command or the like for the camera 100 from a smartphone, a PC, or the like and notifies the camera control unit 220 of the control command. The communication unit 251 is a wireless LAN interface that performs wireless communication, and performs hardware processing of communication performed by the communication I/F unit 250.

The release button 11, the button 12, and the touch panel 13 have the above-described functions, and output operation information (shooting preparation instruction, shooting instruction, display range selection instruction, etc.) according to the operation to the IO controller 270. The IO controller 270 outputs operation information from various operation units to the camera control unit 220, and the camera control unit 220 performs processing according to the operation information from the IO controller 270. For example, the touch panel 13 is overlaid on the display unit 141 and outputs operation information including coordinate information of a position or area where a touch operation is performed.

The tripod joining sensor 290 detects that the tripod is joined to the joining portion on the bottom surface of the camera 100. When the tripod joint sensor 290 detects that the tripod 120 is joined to the joint portion of the camera 100 at the start of photographing, the camera control unit 220 associates information indicating that with the VR image data obtained by photographing. The recording medium 231 is controlled to be recorded.

FIG. 3 shows a flowchart of the operation of the camera 100 in this embodiment. Each process in this flowchart is realized by the camera control unit 220 loading a program recorded in the ROM 280 into the RAM 260 and executing the program. The flowchart of FIG. 3 starts, for example, when the camera 100 is powered on and the shooting mode is set.

The camera control unit 220 acquires the video data output from the image processing unit 208 (S301). The camera control unit 220 receives the VR moving image recording start instruction in response to the user's instruction to the release button 11 (Yes in S302). When the tripod joining sensor 290 detects that the tripod 120 is joined to the camera 100 (Yes in S303), the camera control unit 220 detects a person's face from the video data. The camera control unit 220 determines that the feature amount of the detected face image of the person matches the feature amount of the face image of the photographer registered in advance, and the area of the face image recognized as the photographer has a predetermined size or more. If (Yes in S304), the timer count is started (S305). When the timer count has passed the predetermined time (Yes in S306), the camera control unit 220 subjects the video data to the moving image compression encoding process to generate VR moving image data (S307), and records the VR moving image data in the recording medium (S308). Then, the camera control unit 220 controls not to record the VR moving image data until the timer count reaches the predetermined time (No in S306). On the other hand, if the tripod 120 is not joined to the camera 100 (No in S303), the face image of the photographer is not recognized, or the face image of the photographer is smaller than a predetermined size (No in S304), camera control is performed. The unit 220 immediately executes the processes of S307 and S308 in response to the moving image recording start instruction. Then, the camera control unit 220 repeats the processing of S307 and S308 until the end of the moving image shooting mode is detected (No in S309).

According to the present embodiment, since the VR moving image data starts to be recorded after the photographer moves to the correct position after the photographer has been largely captured, the portion not intended by the photographer is recorded. Not be played.

(Second embodiment)
FIG. 4 shows a flowchart of the operation of the camera 100 in this embodiment. Each process in this flowchart is realized by the camera control unit 220 loading a program recorded in the ROM 280 into the RAM 260 and executing the program. The flowchart of FIG. 4 starts, for example, when the camera 100 is powered on and the reproduction mode is set.

The camera control unit 220 selects any one of the plurality of VR moving image data stored in the recording medium 231 as a reproduction target and selects it according to a user operation (S401). When the reproduction start instruction is received (Yes in S402), the start position of the moving image data is set as the initial value of the reproduction start position (S403). The camera control unit 220 determines whether the tripod 120 is joined to the camera 100 at the time of shooting the selected VR moving image data. Here, if the tripod 120 is joined to the joining portion provided on the bottom surface of the camera 100 at the time of shooting the VR moving image data and is detected by the tripod joining sensor 290, the attribute information indicating that is associated with the VR moving image data. And is recorded on the recording medium 231.

When the tripod 120 is joined to the camera 100 at the time of shooting the selected VR moving image data (Yes in S404), the face of the person is detected from the first frame of the VR moving image data. The camera control unit 220 determines that the feature amount of the detected face image of the person matches the feature amount of the face image of the photographer registered in advance, and the area of the face image recognized as the photographer has a predetermined size or more. If (Yes in S405), a predetermined time is added to the reproduction start position (S406). The camera control unit 220 reproduces the VR moving image data from the set reproduction start position (S407). Then, the camera control unit 220 repeats the processing of S407 until the end of the reproduction mode is detected (No in S408).

It should be noted that the predetermined time is a time predetermined as a time sufficient for the photographer to move to the correct position. The predetermined time may be the time until the area of the face image of the photographer included in the frame of the VR moving image data becomes smaller than the predetermined size.

According to the present embodiment, since the photographer moves to the correct position and the reproduction is started from the portion photographed after the photographer largely appears in the VR moving image data, the portion not intended by the photographer is reproduced. It will not be done.

(Third Embodiment)
FIG. 5 shows a flowchart of the operation of the camera 100 in this embodiment. Each process in this flowchart is realized by the camera control unit 220 loading a program recorded in the ROM 280 into the RAM 260 and executing the program. The flowchart of FIG. 5 starts, for example, when the camera 100 is powered on and the reproduction mode is set. Further, the same operations as those in FIG. 4 are denoted by the same reference numerals, and the description thereof will be omitted.

The camera control unit 220 deletes the portion from the beginning to the predetermined time of the selected VR moving image data (S501), updates the VR moving image data, and records it on the recording medium 231 (S502).

It should be noted that the predetermined time is a time predetermined as a time sufficient for the photographer to move to the correct position. The predetermined time may be the time until the area of the face image of the photographer included in the frame of the VR moving image data becomes smaller than the predetermined size.

According to the present embodiment, since the VR moving image data is re-recorded by deleting the portion in which the photographer is largely captured, the portion not intended by the photographer is not reproduced.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a recording medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Claims (5)

Detection means for detecting that the tripod has been joined,
Determination means for determining whether or not the size of the face image of the photographer recognized from the VR image captured by the image capturing means is larger than a predetermined size;
When the tripod is joined and the size of the face image of the photographer is larger than a predetermined size, the VR image captured by the image capturing means is compression-encoded after a predetermined time has elapsed after receiving the capturing instruction. Generating means for generating VR moving image data by
An electronic device comprising recording means for recording the VR moving image data on a recording medium. A detection means for detecting that the tripod is joined to the image pickup device at the time of shooting the VR moving image to be played back;
Determination means for determining whether or not the size of the face image of the photographer recognized from the VR moving image is larger than a predetermined size;
An electronic device comprising: a reproduction unit that reproduces a VR moving image from a position where a predetermined time has elapsed from the beginning when the size of the face image of the photographer is larger than a predetermined size when the tripod is joined. A detection means for detecting that the tripod is joined to the image pickup device at the time of shooting the VR moving image to be played back;
Determination means for determining whether or not the size of the face image of the photographer recognized from the VR moving image is larger than a predetermined size;
When the tripod is joined and the size of the face image of the photographer is larger than a predetermined size, a deletion unit for deleting a portion from the beginning to a position where a predetermined time has elapsed from the VR moving image is provided. Electronic equipment to do. The electronic device according to claim 2, wherein the predetermined time period is a predetermined time period or a time period until the size of the face image of the photographer becomes smaller than a predetermined size in the VR moving image. machine. A program that causes a computer to function as each unit of the electronic device according to claim 1.

Images