JP2021097280A - Imaging apparatus, image creation method, and program - Google Patents

Abstract

To provide an imaging apparatus that can control the angle of view of an image without requiring predetermined hardware.SOLUTION: An imaging apparatus 5 combines pieces of image data picked up by a plurality of image pick-up devices 21A, 21B to generate image data that has a wider angle than that of image data picked up by one image pick-up device. When the average value of the luminance of the image data picked up by one of the image pick-up devices has periodicity, the imaging apparatus does not use the image data having periodicity for composition, and outputs the image data picked up by the other of the image pick-up devices.SELECTED DRAWING: Figure 2

Description

The present invention relates to an image pickup apparatus, an image generation method, and a program.

There is known an imaging device that uses a plurality of wide-angle lenses such as a fisheye lens and an ultra-wide-angle lens to image a circumference of 360 degrees or all directions at once. In this image pickup apparatus, an image from each lens is projected onto each image pickup device, and the obtained images are joined by image processing to generate a 360-degree image.

When the imager provides an image captured by such an imaging device, it may not be possible to easily specify a desired subject due to the wide angle of view. That is, it is not easy to tell the viewer where in the spherical image of the 360-degree image the subject to be clearly shown appears.

Therefore, a technique for narrowing the angle of view is known (see, for example, Patent Document 1). Patent Document 1 discloses an imaging device that switches between a 360-degree imaging mode and a 180-degree imaging mode at a timing intended by the user by pressing a touch panel or a physical button.

However, the conventional technique has a problem that hardware that accepts a user's operation for narrowing the angle of view of an image is required.

In view of the above problems, an object of the present invention is to provide an imaging device capable of controlling the angle of view of an image without a predetermined hardware.

In view of the above problems, the present invention is an image pickup device that synthesizes image data captured by a plurality of image pickup elements to generate image data having a wider angle than the image data captured by one image pickup device. When the average brightness of the image data captured by the image sensor has periodicity, the characteristic is that the image data captured by the other image sensor is output without using the periodic image data for synthesis. To do.

It is possible to provide an image pickup apparatus capable of controlling the angle of view of an image without a predetermined hardware.

It is a figure explaining the use case of an example of an image communication system. It is a figure explaining the schematic operation of the image communication system that a real estate agent who guides a real estate property displays an arbitrary predetermined range to a customer in a real estate store. This is an example of a schematic diagram of the configuration of an image communication system. It is an example of the schematic diagram of another configuration of an image communication system. This is an example of a cross-sectional view of an image pickup apparatus. It is a figure which shows the hardware configuration example of the image pickup apparatus. This is an example of a hardware configuration diagram of a terminal device. It is a use image figure of the image pickup apparatus. It is a figure explaining the outline of the process from the image captured by the image pickup apparatus to the creation of a 360 degree image (No. 1). It is a figure explaining the outline of the process from the image captured by the image pickup apparatus to the creation of a 360 degree image (No. 2). It is a figure which showed the position of the virtual camera and a predetermined area when the whole celestial sphere image is made into a three-dimensional stereoscopic sphere. This is an example of a functional block diagram showing the functions of the input image processing module in a block shape. It is a figure which shows an example of the space which an image pickup apparatus takes an image. It is a figure which shows an example of the time change of the brightness average value by a gesture switching operation. This is an example of a flowchart showing a procedure in which an imaging device switches from a 360-degree image to a 180-degree image. It is a figure which shows the structural example of the luminance value characteristic storage part. It is a figure which shows an example of the time change of the brightness average value by a gesture switching operation. This is an example of a flowchart showing a procedure in which the imaging device switches from a 360-degree image to a 180-degree image (Example 2). It is a figure which shows the function of the luminance value comparison part (Example 3). It is a figure which shows an example of time change of the brightness average value by a gesture switching operation (Example 3). This is an example of a flowchart showing a procedure in which the imaging device switches from a 360-degree image to a 180-degree image (Example 3). It is a figure which shows the division example of image data. It is a figure which shows the configuration example of the input image processing module (Example 4). This is an example of a flowchart for explaining the processing of the input image processing module. It is a figure which shows the configuration example of the input image processing module (Example 5). It is a figure which shows an example of the time change of the brightness average value by a gesture switching operation (Example 5). This is an example of a flowchart for explaining the processing of the input image processing module (Example 5). It is a figure which shows the structural example of the luminance value comparison part (Example 6). It is an example of the figure explaining the target of comparison between areas. It is a figure which shows an example of the time change of the brightness average value by a gesture switching operation (Example 6). This is an example of a flowchart for explaining the processing of the input image processing module (Example 6). This is an example of a diagram for explaining the configuration of the input image processing module (Example 7). This is an example of a flowchart for explaining the processing of the input image processing module.

Hereinafter, as an example of the embodiment for carrying out the present invention, the image pickup apparatus and the image generation method performed by the image pickup apparatus will be described with reference to the drawings.

<Outline of image communication system>
FIG. 1 is a diagram illustrating a use case of the image communication system 100. The real estate agent 101 in the real estate property 110 grips the image pickup device 5 and moves in each room while capturing a 360-degree image. The image pickup device 5 transmits a 360-degree image (moving image) in real time to the terminal device 10A in the real estate store. The terminal device 10A displays a predetermined range of the received 360-degree image on a display which is a flat device.

The customer 102 in the real estate store can operate the terminal device 10A to rotate the image 360 degrees (display an arbitrary predetermined range), and can browse any direction of the real estate property while staying in the real estate store.

The case where the real estate agent 101 in the real estate property 110 wants to show the customer 102, for example, the air conditioner installed in the room will be described. In the past, the real estate agent 101 had to explain by voice where the air conditioner was shown in the 360-degree image, but there was a risk that communication would not be easy and the explanation would be cumbersome.

Therefore, the image communication system 100 of the present embodiment makes it possible to easily display the subject that the real estate agent 101 wants to show on the terminal device 10A operated by the customer 102 as follows.

The schematic operation of the image communication system 100 will be described with reference to FIG. FIG. 2 is a diagram illustrating a schematic operation of an image communication system 100 in which a real estate agent 101 in a real estate property 110 displays an arbitrary predetermined range on a customer 102 in a real estate store 120.

The image pickup device 5 held by the real estate agent 101 has two or more image pickup elements because it can capture a 360-degree image. In the present embodiment, the image pickup device 5 having two image pickup elements will be mainly described, but the number of image pickup elements may be three or more. The image pickup device 5 creates a 360-degree image by stitching (synthesizing) the image data captured by the two image pickup elements.

When the real estate agent 101 wants to explain the air conditioner 901 by focusing the customer 102 on the air conditioner 901, for example, the real estate agent 101 hands in front of the image sensor (the image sensor that does not show what he wants to capture) on the opposite side of the air conditioner 901. Operate periodically. Hereinafter, this operation is referred to as "gesture switching operation". When the real estate agent 101 moves his / her hand left / right (or up / down) so as to pass over the lens of the image pickup device 5 many times, the brightness changes periodically. This is because the area covered by an object on one lens and the detected brightness average value are inversely proportional to each other. In addition, for example, gestures such as moving the hand closer / away and opening / closing the hand make the same change.

In FIG. 2, the lens on the opposite side of the air conditioner 901 is periodically shielded from light by the palm of the hand. When the image pickup device 5 detects that the brightness detected by at least one of the two image pickup elements has changed periodically, the image data captured by the two image pickup elements is not combined and the brightness changes. Only the image data of the one that does not exist is transmitted to the terminal device 10A of the real estate store 120. In the present embodiment, a 360-degree (all-sky) image can be obtained by synthesizing the image data captured by the two image sensors, respectively. Therefore, the image data captured by one image sensor is referred to as a 180-degree image. A combination of two image data captured by two image sensors is called a 360-degree image. The image pickup device 5 maintains the same state (outputs a 180-degree image) until the next gesture switching operation is detected.

The image pickup device 5 may transmit only the image data whose brightness has changed to the terminal device 10A of the real estate store 120. For example, the user-friendly one may be selected and set whether to transmit the image data whose brightness has changed or only the image data whose brightness has not changed.

The terminal device 10A of the real estate store 120 displays the entire 180-degree image, but since the real estate agent 101 takes an image so that the air conditioner 901 is captured at the center of the optical axis of the image pickup device 5, the display of the terminal device 10A is displayed. The air conditioner 901 is displayed near the center. Therefore, the customer 102 does not need to adjust the display range so that the air conditioner 901 is captured.

In this way, the image communication system 100 of the present embodiment can switch from a 360-degree image to a 180-degree image and display an arbitrary range from a wide-angle image. In addition, it can be executed without using a predetermined hardware switch such as a touch panel or a physical key, and can be realized by an intuitive operation. Therefore, the customer 102 can browse the subject that the real estate agent 101 wants to pay attention to without explaining where the subject that the real estate agent 101 wants to pay attention to.

<Terminology>
Synthesis means combining two or more things into one. In the present embodiment, it means to generate wide-angle image data with two or more images.

Periodicity means that the same situation occurs after a certain period of time. However, it does not have to be strict to be constant.

<Outline of image communication system>
Subsequently, the outline of the configuration of the image communication system 100 of the present embodiment will be described with reference to FIG. FIG. 3 is a schematic diagram of the configuration of the image communication system 100 of the present embodiment.

As shown in FIG. 3A, in the image communication system 100 of the present embodiment, the terminal device 10A arranged at the base A and the image pickup device 5 arranged at the base B connect to a communication network N such as the Internet or a cellular line. Can communicate via. The base A is, for example, a real estate store 120 having a customer 102, and the base B is, for example, a real estate property 110 to be previewed.

As described above, the imaging device 5 is a special digital camera that captures a subject, a landscape, or the like and synthesizes two 180-degree images that are the basis of the 360-degree image.

The terminal device 10A is a general-purpose information processing device that communicates with other bases by operating application software for video conferencing. The terminal device 10A is, for example, a notebook PC (Personal Computer), a mobile phone, a smartphone, a tablet terminal, a car navigation system, a game machine, a PDA (Personal Digital Assistant), a wearable PC, a desktop PC, or the like. However, it is not limited to these.

The terminal device 10A displays the received image on the display 720 provided in the own device.

In addition, OpenGL ES (Open Graphics Library for Embedded Systems) is installed in the terminal device 10A, and a predetermined area image can be created from the 360-degree image sent from the image pickup device 5. Therefore, each terminal device 10A can display a predetermined region image cut out from the 360-degree image.

At least one customer 102 exists at the base A, and the terminal device 10A displays an image (360-degree image or 180-degree image) captured by the image pickup device 5 on the display 720. The customer 102 of the base A is a user who browses a 180-degree image or a 360-degree image switched by the real estate agent 101. However, the number of customers 102 is an example.

At the base B, at least one real estate agent 101 exists, and holds the image pickup device 5 by holding it in his hand or attaching it to a rod-shaped member. The real estate agent 101 can move together with the image pickup device 5. The real estate agent 101 is an imager who images the real estate property 110. However, the number of real estate agents 101 is an example.

Next, as shown in FIG. 3B, in another image communication system 100 of the present embodiment, at least two bases A and terminal devices 10A and 10B arranged at the bases B are communication networks N such as the Internet. The terminal devices 10A and 10B of each base can share the video by communicating with each other. The image pickup device 4 and the terminal device 10A are arranged at the base A, and the terminal device 10B and the image pickup device 5 are arranged at the base B. The base A is, for example, a real estate store 120 having a customer 102, and the base B is, for example, a real estate property 110 to be previewed.

As described above, the image pickup device 5 is a special digital camera that captures a subject, a landscape, or the like and synthesizes two 180-degree images that are the basis of the 360-degree image. On the other hand, the image pickup device 4 is a general digital camera for capturing a subject, a landscape, or the like to obtain a general flat image.

The terminal devices 10A and 10B are general-purpose information processing devices that communicate with other bases by operating application software for video conferencing. The terminal devices 10A and 10B are, for example, a notebook PC (Personal Computer), a mobile phone, a smartphone, a tablet terminal, a car navigation system, a game machine, a PDA (Personal Digital Assistant), a wearable PC, a desktop PC, or the like. However, it is not limited to these.

The terminal device 10A displays an image of a video call on a display 720 provided in the terminal device 10A. The terminal device 10A uses an external imaging device 4 to image the customer 102 and the like.

The terminal device 10B displays an image of a video call on a display 720 provided in the terminal device 10B. The terminal device 10B usually takes an image with a CMOS sensor 712 (Complementary Metal Oxide Semiconductor) or the like, which will be described later, provided in the own device, but uses wireless communication technology such as Wi-Fi (Wireless Fidelity) or Bluetooth (registered trademark). Then, the 360-degree image obtained by the image pickup apparatus 5 can be acquired.

In addition, OpenGL ES is installed in the terminal devices 10A and 10B, and a predetermined area image showing a part of the 360-degree image can be created, or a 360-degree image sent from another terminal device 10 can be created. A predetermined area image can be created from. Therefore, each of the terminal devices 10A and 10B can display a predetermined area image cut out from the 360-degree image.

At least one customer 102 exists at the base A, and the terminal device 10A displays an image (360-degree image or 180-degree image) captured by the image pickup device 5 and an image captured by the image pickup device 4 on the display 720. The customer 102 of the base A is a user who browses a 180-degree image or a 360-degree image switched by the real estate agent 101. However, the number of customers 102 is an example.

At the base B, at least one real estate agent 101 exists, and holds the image pickup device 5 by holding it in his hand or attaching it to a rod-shaped member. The real estate agent 101 can move together with the image pickup device 5. The real estate agent 101 is an imager who images the real estate property 110. Further, the terminal device 10B displays the images captured by the image pickup devices 4 and 5 on the display 720. However, the number of real estate agents 101 is an example.

The communication management system 80 manages and controls the communication of the terminal devices 10A and 10B. Therefore, the communication management system 80 is also a communication control system. The communication management system 80 is installed in a data center or cloud that provides communication services. Further, the communication management system 80 may be constructed by a single computer, or may be constructed by a plurality of computers arbitrarily assigned by dividing each unit (function, means, or storage unit).

The number of bases, the types of terminal devices 10A and 10B arranged at each base, the types of image pickup devices 4 and 5, and the number of users (customers and real estate agents) shown in FIG. 3B are examples. In the embodiment, it is sufficient that the base A and one other base are provided, but the image communication system 100 can communicate at three or more bases. Further, the base A does not have to have the image pickup device 4, and it is sufficient that the base A can display a 360-degree image transmitted from the base B.

<Configuration example>
Hereinafter, the overall configuration of the image pickup apparatus 5 according to the present embodiment will be described with reference to FIGS. 4 and 5. FIG. 4 is a cross-sectional view of the image pickup apparatus 5 according to the present embodiment. The image pickup apparatus 5 shown in FIG. 4 includes an image pickup body 19, a housing 17 that holds components such as a control unit 16 and a battery 15 in addition to the image pickup body 19, and an image pickup button 18 provided on the housing 17. To be equipped.

The image pickup body 19 shown in FIG. 4 has two imaging optical systems 20A and 20B, and an image pickup element 21A and an image pickup element 21B. The image pickup devices 21A and 21B (an example of the first image pickup device and the second image pickup device, respectively) are a CCD (Charge Coupled Device) sensor, a CMOS sensor, and the like. The imaging optical systems 20A and 20B are configured as, for example, a fisheye lens having 7 elements in 6 groups. In the embodiment shown in FIG. 4, the fisheye lens has an angle of view larger than 180 degrees (= 360 degrees / n; the number of optical systems n = 2), and preferably has an angle of view of 190 degrees or more. A combination of such wide-angle imaging optical systems 20A and 20B and image pickup elements 21A and 21B one by one is referred to as a wide-angle imaging optical system.

The optical elements (lens, prism, filter and aperture stop) of the two image pickup optical systems 20A and 20B are positioned with respect to the image pickup elements 21A and 21B. The optical axis of the optical elements of the imaging optical systems 20A and 20B is located orthogonal to the central portion of the light receiving region of the corresponding image sensor 21, and the light receiving region serves as the imaging surface of the corresponding fisheye lens. Positioning is performed so as to.

In the embodiment shown in FIG. 4, the imaging optical systems 20A and 20B have the same specifications, and are combined in opposite directions so that their optical axes match. The image pickup devices 21A and 21B convert the received light distribution into an image signal, and sequentially send an image frame to the control unit 16 (a frame is image data, but refers to one of continuous image data. Often used in cases) is output. The images captured by the image sensors 21A and 21B are combined, whereby an image having a solid angle of 4π steradian (360-degree image or spherical image) is generated. The 360-degree image is an image taken in all directions that can be seen from the imaging point. In the embodiment described, it is described as generating a 360-degree image, but it may be a so-called panoramic image in which only the horizontal plane is imaged at 360 degrees, and a part of the whole view of the whole celestial sphere or the horizontal plane of 360 degrees is taken. It may be an captured image. Further, the 360-degree image can be saved as a still image or as a moving image.

FIG. 5 is a diagram showing a hardware configuration example of the image pickup apparatus 5. The image pickup device 5 includes a CPU 30, a main memory 31, a flash ROM 32, a fisheye lens 14A, 14B, an image sensor 21A, 21B, an A / D conversion circuit 33A, 33B, an input image processing module 34A, 34B, a 3-axis acceleration sensor 35, and distortion correction. It has an image composition module 36, an image codec 37, an audio codec 38, a microphone 39, a speaker 40, a wireless communication module 41, a memory card I / F module 42, and a memory card 43.

The CPU 30 expands and executes the program stored in the flash ROM 32 in the main memory 31 to control the entire image pickup apparatus 5. The main memory 31 is, for example, a DRAM (Dynamic Random Access Memory) and is used as a work area of the CPU 30. The flash ROM 32 stores various programs for starting the system when the power is turned on and realizing the functions of the image pickup apparatus 5.

The image codec 37 compresses the image data. The audio codec 38 compresses the audio data collected by the microphone, decompresses the audio data input from the wireless communication module 41, and outputs the audio data from the speaker 40.

The wireless communication module 41 is connected to a cellular line to wirelessly transmit and receive video, audio, and the like. The memory card I / F module 42 writes data to the mounted memory card 43, and reads data from the memory card 43.

The analog image signals output from the image pickup elements 21A and 21B are converted into digital image data by the A / D conversion circuits 33A and 33B, and these image data are input to the input image processing modules 34A and 34B.

The input image processing modules 34A and 34B use the brightness information of the image data to measure light and adjust the exposure amount by adjusting the exposure control (AE (Automatic Exposure)) and making the ratio of RGB uniform to improve the color reproducibility. The white balance (AWB (Automatic White Balance)) for this purpose is processed. The image data subjected to these processes is input to the distortion correction / image composition module 36.

The distortion correction / image composition module 36 uses the information from the 3-axis acceleration sensor 35 to perform distortion correction and top-bottom correction on the two image data to synthesize a 360-degree image. The 360-degree image data is compressed by the image codec 37 and transmitted to the terminal device 10A in the real estate store 120 by wireless communication (cellular line). Further, the audio signal collected by the microphone 39 is also transmitted to the terminal device 10A in the real estate store 120 by wireless communication (cellular line).

FIG. 6 is a hardware configuration diagram of the terminal device 10. As shown in FIG. 6, the terminal device 10 includes a CPU 701, a ROM 702, a RAM 703, a flash memory 704, an SSD 705, a media I / F 707, an operation button 708, a power switch 709, a bus line 710, a network I / F 711, and a CMOS. The sensor 712, the image sensor I / F713, the microphone 714, the speaker 715, the sound input / output I / F716, the display I / F717, the external device connection I / F718, the short-range communication circuit 719, and the antenna 719a of the short-range communication circuit 719 are provided. ing. Of these, the CPU 701 controls the operation of the entire terminal device. The ROM 702 stores a program used for booting an OS such as an IPL. The RAM 703 is used as a work area for the CPU 701. The flash memory 704 stores various data such as a communication program, image data, and sound data. The SSD 705 controls reading or writing of various data to the flash memory 704 according to the control of the CPU 701. An HDD may be used instead of the SSD. The media I / F 707 controls reading or writing (storage) of data to a recording medium 706 such as a flash memory. The operation button 708 is a button that is operated when selecting a destination of the terminal device 10. The power switch 709 is a switch for switching ON / OFF of the power supply of the terminal device 10.

Further, the network I / F711 is an interface for data communication using a communication network N such as the Internet. The CMOS sensor 712 is a kind of built-in imaging means that images a subject and obtains image data under the control of the CPU 701. It should be noted that the imaging means such as a CCD sensor may be used instead of the CMOS sensor. The image sensor I / F 713 is a circuit that controls the drive of the CMOS sensor 712. The microphone 714 is a built-in device that converts sound into an electrical signal. The speaker 715 is a built-in device that converts an electric signal into physical vibration to produce sounds such as music and voice. The sound input / output I / F 716 is a circuit that processes sound signal input / output between the microphone 714 and the speaker 715 under the control of the CPU 701. The display I / F 717 is a circuit that transmits image data to an external display under the control of the CPU 701. The external device connection I / F718 is an interface for connecting various external devices. The short-range communication circuit 719 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark).

Further, the bus line 710 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 701 shown in FIG.

The display 720 is a kind of display means composed of a liquid crystal, an organic EL (Electro Luminescence), or the like for displaying an image of a subject, an operation icon, or the like. Further, the display 720 is connected to the display I / F 717 by a cable. This cable may be a cable for analog RGB (VGA) signals, a cable for component video, HDMI (High-Definition Multimedia Interface) (registered trademark), or DVI (Digital Video Interactive). ) It may be a signal cable.

The CMOS sensor 712 is a kind of built-in imaging means for capturing an image of a subject and obtaining image data under the control of the CPU 701. It should be noted that the imaging means such as a CCD sensor may be used instead of the CMOS sensor. External devices such as an external camera, an external microphone, and an external speaker can be connected to the external device connection I / F718 by a USB cable or the like. When an external camera is connected, the external camera is driven in preference to the built-in CMOS sensor 712 according to the control of the CPU 701. Similarly, when an external microphone is connected or an external speaker is connected, the external microphone takes precedence over the built-in microphone 714 and the built-in speaker 715, respectively, according to the control of the CPU 701. And an external speaker drives.

Further, the recording medium 706 has a structure that can be attached to and detached from the terminal device 10. Further, as long as it is a non-volatile memory that reads or writes data under the control of the CPU 701, not only the flash memory 704 but also an EEPROM or the like may be used.

<How to generate a 360-degree image>
Next, a method of generating a 360-degree image will be described with reference to FIGS. 7 to 9. FIG. 7 is an image diagram of use of the image pickup apparatus 5. As shown in FIG. 7, the image pickup apparatus 5 is used, for example, for a user (a real estate agent in the present embodiment) to hold the image in his / her hand and take an image of a subject around the user. In this case, two 180-degree images can be obtained by taking an image of a subject around the user by the image sensor 21A and the image sensor 21B shown in FIG. 4, respectively.

Next, the outline of the process from the image captured by the image pickup apparatus 5 to the creation of the 360-degree image will be described with reference to FIGS. 8 and 9. 8 (a) is a 180-degree image (front side) captured by the image pickup device 5, FIG. 8 (b) is a 180-degree image (rear side) captured by the image pickup device 5, and FIG. 8 (c) is positive. It is a figure which showed the image represented by the equirectangular projection (hereinafter, referred to as "equirectangular projection image"). FIG. 9A is a conceptual diagram showing a state of covering a sphere in an equirectangular projection image, and FIG. 9B is a diagram showing a 360-degree image.

As shown in FIG. 8A, the image obtained by the image sensor 21A is a 180-degree image (front side) curved by the fisheye lens 14A. Further, as shown in FIG. 8B, the image obtained by the image sensor 21B is a 180-degree image (rear side) curved by the fisheye lens 14B. Then, the 180-degree image (front side) and the 180-degree image (rear side) inverted 180 degrees are combined by the image pickup apparatus 5, and as shown in FIG. 8 (c), the equirectangular projection image. Is created.

Then, by using OpenGL ES in the terminal device 10, as shown in FIG. 9A, the equirectangular projection image is pasted so as to cover the spherical surface, and is shown in FIG. 9B. A 360 degree image as shown above is created. As described above, the 360-degree image is represented as an image in which the equirectangular projection image faces the center of the sphere. OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. The 360-degree image may be a still image or a moving image.

As described above, since the 360-degree image is an image pasted so as to cover the spherical surface, it gives a sense of discomfort to humans. Therefore, by displaying a part of a predetermined area of the 360-degree image (hereinafter, referred to as "predetermined area image") as a flat image with less curvature, it is possible to display the image without giving a sense of discomfort to humans.

FIG. 10 is a diagram showing the positions of a virtual camera and a predetermined area when the spherical image is a three-dimensional stereoscopic sphere. The virtual camera IC corresponds to the position of the viewpoint of the user who views the image with respect to the spherical image CE displayed as a three-dimensional stereoscopic sphere.

As shown in FIG. 10, the virtual camera IC is located inside the spherical image CE. A predetermined region T in the spherical image CE is an imaging region of the virtual camera IC. The zoom of the predetermined region T can be expressed by expanding or contracting the range (arc) of the angle of view α. Further, the zoom of the predetermined area T can be expressed by moving the virtual camera IC closer to or further away from the spherical image CE. The predetermined region image Q is an image of the predetermined region T in the spherical image CE.

<About functions>
FIG. 11A is an example of a functional block diagram showing the functions of the input image processing modules 34A and 34B in a block shape. It is assumed that the functions of the input image processing modules 34A and 34B are the same, but there may be a function existing in only one of them. Further, FIG. 11B is a diagram showing in detail the function of the luminance value comparison unit 52.

The input image processing modules 34A and 34B include an exposure control unit 50, a brightness value storage unit 51, a brightness value comparison unit 52, a white balance control unit 53, a gamma correction unit 54, and a color conversion unit 55. The exposure control unit 50 measures light using the luminance information of the image data output from the image sensors 21A and 21B and converted to A / D, adjusts the exposure amount, and adjusts the exposure amount, and the 360-degree image and the 180-degree image described later. Performs processing related to switching. The exposure control unit 50 calculates the brightness average value for each frame.

The brightness value storage unit 51 temporarily stores the brightness value obtained by the exposure control unit 50 in order to compare it with the image data later in time. Store image data for at least a period of time that allows the periodicity to be determined. The brightness value comparison unit 52 detects a periodic change in the brightness value based on the brightness values for a plurality of past frames stored in the brightness value storage unit 51.

The white balance control unit 53 performs a process of making the ratio of RGB uniform in order to improve the color reproducibility. The gamma correction unit 54 performs a process of adjusting the hue of the entire image for one frame. The color conversion unit 55 converts the RGB digital image data into a color difference component (Cb, Cr) and a luminance component (Y).

The processing when the amount of light input from the fisheye lenses 14A and 14B changes and a specific periodicity is detected in the change pattern of the brightness average value input to the input image processing modules 34A and 34B over a plurality of frames will be described. The processing contents will be described with reference to FIG. 13).

The exposure control unit 50 of the input image processing modules 34A and 34B obtains the average value of the brightness component values of all the image data for one input frame, and stores the brightness average value in the brightness value storage unit 51. This is repeated for a plurality of frames, and the average luminance value is sequentially stored in the luminance value storage unit 51 in accordance with the passage of time.

In parallel with the processing of the luminance value storage unit 51, the luminance value comparison unit 52 compares the difference between the temporally adjacent luminance average values stored in the luminance value storage unit 51. As shown in FIG. 11B, the luminance value comparison unit 52 includes a luminance difference calculation unit 60, a luminance value feature storage unit 61, and a luminance value feature determination unit 62.

First, the luminance difference calculation unit 60 calculates the luminance difference between the luminance average value of the processing target frame (current processing target frame) and the luminance average value of the previous frame. The luminance difference calculation unit 60 stores the luminance difference in the luminance value feature storage unit 61. In the luminance value feature determination unit 62, is the difference between the previous luminance difference stored in the luminance value feature storage unit 61 and the current luminance difference equal to or greater than a preset threshold value A (an example of the first threshold value)? Judge whether or not. When the threshold value A or more is set, the frame number of the processing target frame (corresponding to the time when the processing target frame is acquired) is stored in the luminance value feature storage unit 61. Then, the interval from the frame number stored last time as the difference in brightness difference is equal to or greater than the threshold value A (that is, the time interval) is less than the preset threshold value B (an example of the second threshold value). In this case, the luminance value feature determination unit 62 determines that the luminance average value for each frame has periodicity.

Then, when it is determined that there is periodicity, the input image processing modules 34A and 34B output a gesture switching signal to the distortion correction / image composition module 36. When the luminance value feature determination unit 62 again determines that the luminance average value for each frame has periodicity, the input image processing modules 34A and 34B output a gesture switching signal to the distortion correction / image composition module 36. Therefore, the user can switch between the 180-degree image and the 360-degree image (the angle of view can be controlled) by the same gesture switching operation.

<Relationship between imaging target and imaging device 5>
Based on FIG. 12, the operation of the real estate agent 101 and the processing of the imaging device 5 when switching from the 360-degree image to the 180-degree image will be described. FIG. 12 is a diagram showing an example of the positions of the image pickup target and the image pickup device 5.

The real estate agent 101, who is guiding the real estate property, holds the monopod 3 and takes an image of the room with the image pickup device 5, and explains to the customer 102 in the real estate store by voice. First, the image of the whole celestial sphere is transmitted so that the customer 102 can see the entire room (space A and space B). When explaining the air conditioner 901 installed in the room, the position of the air conditioner 901 in the 360-degree image. Need to be explained.

In order to omit the description of the position of the air conditioner 901, the real estate agent 101 switches the 360-degree image to the 180-degree image, and adjusts the orientation and tilt of the image pickup device 5 so that the air conditioner 901 appears near the center of the 180-degree image. adjust.

The space imaged by the image sensor 21A is defined as the A space, and the space imaged by the image sensor 21B is defined as the B space. Now, when the air conditioner 901 is in the A space, the real estate agent 101 changes the direction and inclination of the image pickup device 5 so that the optical axis of the fisheye lens 14A is approximately oriented toward the air conditioner 901. Then, when the hand held over the fisheye lens 14B is shaken several times, the amount of light input from the fisheye lens 14B periodically decreases. When the luminance value comparison unit 52 of the input image processing module 34B detects a periodic decrease, it outputs a gesture switching signal to the distortion correction / image composition module 36. When the distortion correction / image composition module 36 detects this gesture switching signal, the 360-degree image composition process is stopped, and the image data input from the input image processing module 34A (the input image processing module that does not output the gesture switching signal). ), Switch to the mode to generate a 180 degree image by performing top-bottom correction together with distortion correction. This 180-degree image is compressed to H.264 by the image codec 37, and the wireless communication module 41 transmits this image data to the terminal device 10A in the real estate store 120 using the communication network N (cellular line or the like). To do. On the display of the terminal device 10A, the air conditioner 901 is displayed near the center of the 180-degree image.

Even if the real estate agent 101 stops waving his hand over the fisheye lens 14B, the image pickup device 5 continues to generate a 180-degree image.

After that, when the real estate agent 101 in the real estate property 110 waved his hand over the fisheye lens 14B again, the amount of light input from the fisheye lens 14B periodically decreases. When the luminance value comparison unit 52 of the input image processing module 34B detects a periodic decrease, it outputs a gesture switching signal to the distortion correction / image composition module 36.

When the distortion correction / image composition module 36 detects this gesture switching signal, it is currently generating a 180-degree image, so that the 360-degree image composition process is restarted. Then, this 360-degree image is compressed to H.264 and transmitted to the terminal device 10A in the real estate store 120 by the wireless communication module 41 using the communication network N (cellular line or the like). If a 180-degree image is currently being generated, the distortion correction / image composition module 36 resumes the 360-degree image composition process regardless of whether the fisheye lens 14A or 14B transmits the gesture switching signal. Good. Even if the user forgets which fisheye lenses 14A and 14B are effective, the synthesis process can be restarted.

When the real estate agent 101 waved his hand over the fisheye lens 14A, the distortion correction / image compositing module 36 may stop using the 180 degree image of the fisheye lens 14B for compositing.

Here, the case where there are two image sensors 21 has been described, but when there are three or more image sensors, when the input image processing module 34 of one image sensor outputs a gesture switching signal, the distortion correction / image synthesis module 36 remains. The image data of the two image sensors 21 of the above are combined. The same applies when the number of image sensors is four or more.

In this way, switching from a 360-degree image to an image captured by the image sensor 21A (approximately 180-degree image) and from a 360-degree image to an image captured by the image sensor 21B (approximately 180-degree image). It is possible to switch between the two by an intuitive operation such as waving a hand over one fisheye lens without using a hard or soft switch. Switching from images captured by the image sensors 21A and 21B (approximately 180-degree images) to 360-degree images is intuitive, such as waving your hand over one fisheye lens without using a hard or soft switch. It can be realized by various operations.

<Detection of periodic changes in brightness value>
In the present embodiment, the 360-degree image and the 180-degree image are switched only when the brightness value comparison unit 52 detects the “specific time change pattern of the brightness average value” one or more times as described above. As a result, not only hardware such as a touch panel and physical keys is not required, but also the following effects can be expected.
-Once switched, the imaging mode (mode for capturing a 360-degree image or a 180-degree image) can be continued until the next switching detection occurs.
-It is possible to prevent switching due to an unintended momentary change in brightness due to fluctuations in brightness.

FIG. 13 shows an example of the time change of the average luminance value due to the gesture switching operation. FIG. 13A is a scatter plot in which the horizontal axis is the time or frame number and the vertical axis is the luminance value, and FIG. 13B is a diagram illustrating a time when the difference in luminance difference is equal to or greater than the threshold value A.

As shown in FIG. 13A, when the real estate agent moves the hand left and right (or up and down, etc.) so as to pass over the lens many times, the brightness changes at a cycle corresponding to the speed of waving the hand. This is because the area covered by an object on one lens and the detected brightness average value are inversely proportional to each other. Further, for example, gestures such as moving the hand closer to or further away from the hand and opening or closing the hand cause the same change.

In FIG. 13A, it is considered that the timing at which the brightness becomes extremely small (takes a downward extremum) appears twice during the period in which the hand makes one round trip over the lens. The luminance value feature determination unit 62 can determine whether it is a gesture or a luminance fluctuation by comparing the length of the period T during which the luminance becomes small with the threshold value B. In FIG. 13B, the difference in luminance difference is equal to or greater than the threshold value A at times t1 to t4. Specifically, the difference in luminance difference is equal to or greater than the threshold value A. The brightness difference itself may be compared with the threshold value A. Further, since it is detected that the frame interval (time) at which the difference in luminance difference becomes equal to or greater than the threshold value A is less than the threshold value B, the luminance value comparison unit 52 detects the gesture switching operation.

The gesture switching operation may be detected when it is detected twice or more that the frame interval (time) at which the difference in luminance difference is equal to or greater than the threshold value A is less than the threshold value B (corresponding to the reciprocation of the hand). .. It is possible to suppress the detection of the gesture switching operation even though it is not the gesture switching operation.

<Operation procedure>
FIG. 14 is an example of a flowchart showing a procedure in which the image pickup apparatus 5 switches from a 360-degree image to a 180-degree image. The process of FIG. 14 is repeatedly executed while the image pickup apparatus 5 synthesizes the two image data to create a 360-degree image.

In the brightness value comparison unit 52 of the input image processing modules 34A and 34B, the exposure control unit 50 stores the average value (luminance average value) of the brightness components of all the pixels in one frame in the brightness value storage unit 51 (S11).

Next, the luminance difference calculation unit 60 calculates the luminance difference between the luminance average value of the frame to be processed and the luminance average value of the previous frame, and stores the luminance difference in the luminance value feature storage unit 61 (S12).

The luminance value feature determination unit 62 determines whether or not the absolute value of the difference between the latest luminance difference stored in the luminance value feature storage unit 61 and the previous luminance difference is the threshold value A or more (S13). If it is not equal to or higher than the threshold value A, the process of FIG. 14 ends.

When the threshold value is A or more, the luminance value feature determination unit 62 stores the frame number of the frame to be processed in the luminance value feature storage unit 61 (S14).

Then, the luminance value feature determination unit 62 determines whether or not the difference between the frame number of the frame to be processed and the frame number stored immediately before is less than the threshold value B (S15). Time can be compared by comparing frame numbers. If it is not less than the threshold value B, the process of FIG. 14 ends.

If it is less than the threshold value B, the luminance value feature determination unit 62 detects the gesture switching operation and outputs a gesture switching signal (S16).

As a result, the distortion correction / image composition module 36 stops the composition processing of the 360-degree image, and generates a 180-degree image for the image input from the other input image processing module 34 (the one in which the gesture switching operation is not detected) (the one in which the gesture switching operation is not detected). S17).

Then, the image codec 37 compresses the 180-degree image into H.264 and transmits it to the terminal device 10A using the communication network N (S18).

To return to the 360-degree image generation, the real estate agent may perform the same gesture switching operation.

<Main effect>
The image communication system 100 of the present embodiment can switch from a 360-degree image to a 180-degree image to display an arbitrary range from a wide-angle image. In addition, it can be executed without using a predetermined hardware switch such as a touch panel or a physical key, and can be realized by an intuitive operation. Once switched, it can continue in either imaging mode until the next gesture switching operation is detected. It is possible to prevent switching due to an unintended momentary change in brightness due to brightness fluctuation or the like.

In this embodiment, the image pickup apparatus 5 capable of reducing the capacity of the brightness value storage unit 51 required for storing the brightness average value will be described. When the image pickup apparatus 5 detects a plurality of periodic reductions in the luminance average value and detects a gesture switching operation, the luminance value storage unit 51 stores the luminance average values for the plurality of cycles. Therefore, for example, when the periodic decrease in the luminance value is detected a plurality of times, the capacity of the luminance value storage unit 51 can be reduced when the gesture switching operation is detected.

<About the configuration of the luminance value feature storage unit>
In this embodiment, the hardware configuration diagrams of FIGS. 4 to 6 and the functional block diagram shown in FIG. 11 described in the above embodiment will be described as being applicable.

FIG. 15 is a diagram showing a configuration example of the luminance value feature storage unit 61 of this embodiment. The luminance value feature storage unit 61 includes a feature compression unit 70 and a feature storage unit 71. The luminance difference data calculated by the luminance difference calculation unit 60 is temporarily stored in the feature compression unit 70, and when the feature compression unit 70 can determine that the luminance difference can be converted into periodic data, the periodic data is used as a parameter. The represented periodic data parameters are stored in the feature storage unit 71. The periodic data parameters stored in the feature storage unit 71 are transferred to the luminance value storage unit 51.

<Periodic data of average brightness value>
FIG. 16 shows an example of the time change of the average luminance value due to the gesture switching operation. Feature The compression unit 70 compresses the data capacity held by the luminance value storage unit 51 by sequentially converting the luminance average value one cycle before into periodic data parameters.

In the current period 301 of FIG. 16, since periodicity is detected, it is necessary to maintain all the average luminance values for each frame. However, since the periodicity detection has been completed for the previous period 302 including the previous period 302, it is sufficient that only the period data parameters (the time when the brightness difference becomes large and the value of the brightness difference) can be retained. By using these, the retained data of the luminance value storage unit 51 can be compressed as follows.
(i) Number of data before compression Number of frames in one cycle x (average brightness + time) → Number of data 20 to 30
(ii) Number of compressed data (number of periodic data parameter data)
Difference in brightness difference above threshold A + time when this brightness difference occurred) x 2 → Number of data 4 <Operation procedure>
FIG. 17 is an example of a flowchart showing a procedure in which the image pickup apparatus 5 switches from a 360-degree image to a 180-degree image. In the description of FIG. 17, the difference from FIG. 14 will be mainly described.

In the process of FIG. 17, step S15-2 is added. In step S15-2, with respect to the brightness average value accumulated in the brightness value storage unit 51, the feature compression unit 70 detects the difference in brightness difference of the threshold value A or more and the difference in brightness (even in the frame number). Good) is stored in the feature storage unit 71 for one cycle. Since it is for one cycle, four data are stored as shown in (ii). The feature storage unit 71 transfers the periodic data parameters to the luminance value feature storage unit 61, and deletes the luminance average value for one cycle stored in the luminance value feature storage unit 61.

The luminance value feature determination unit 62 refers to the periodic data parameters of the luminance value storage unit 51, and detects, for example, a gesture switching operation when detecting periodic data parameters for N cycles (S16-2). Subsequent processing may be the same as in FIG.

<Main effect>
According to this embodiment, in addition to the effect of the first embodiment, the capacity of the average luminance value stored in the luminance value storage unit 51 can be reduced.

In the present embodiment, the image pickup device 5 capable of improving the detection accuracy of the gesture switching operation even when the periodic change of the average luminance value does not appear remarkably even though the user is performing the gesture switching operation. explain.

<Configuration example of brightness value comparison unit>
In this embodiment, the hardware configuration diagrams of FIGS. 4 to 6 described in the first embodiment and the functional block diagrams of the input image processing modules 34A and 34B shown in FIG. 11A can be used. explain.

FIG. 18 is a diagram showing the functions of the luminance value comparison unit 52 of this embodiment. The brightness value storage unit 51 included in the input image processing module 34A is 51A, and the brightness value storage unit 51 included in the input image processing module 34B is 51B. The luminance value storage unit 51A stores the luminance average value calculated from the image data (an example of the first image data) generated by the input image processing module 34A. The luminance value storage unit 51B stores the luminance average value calculated from the image data (an example of the second image data) generated by the input image processing module 34B.

Further, the luminance value comparison unit 52A of the input image processing module 34A includes a luminance difference calculation unit 60A (an example of the first luminance difference calculation unit), a luminance value feature storage unit 61A, and a luminance value feature determination unit 62A (first). The luminance value comparison unit 52B of the input image processing module 34B includes a luminance difference calculation unit 60B (an example of a second luminance difference calculation unit), a luminance value feature storage unit 61B, It also has a luminance value feature determination unit 62B (an example of a second luminance value feature determination unit).

The functions of the luminance value comparison units 52A and 52B of this embodiment are the same as those of the first embodiment, but the luminance average values output by the other input image processing modules 34A and 34B are also input to the luminance difference calculation units 60A and 60B. Will be done. Therefore, the brightness difference calculation units 60A and 60B can calculate not only the brightness difference in the own lens but also the brightness difference between the brightness average value of the other lens and the brightness average value of the own lens. By calculating the relative luminance difference, it becomes possible to detect the feature of the luminance difference in more detail.

<Periodic data of average brightness value>
FIG. 19 shows an example of the time change of the average luminance value due to the gesture switching operation. In FIG. 19, the average brightness value 310B from the other lens acquired at the same time together with the time change of the average brightness value 310A (represented by “○”) from the lens in which the user is performing the gesture switching operation. The time change (represented by "△") is also displayed. The larger change in the average brightness value is the average brightness value 310A of the lens on which the gesture switching operation is performed, and the smaller change is the average brightness value 310B of the lens on which the gesture switching operation is not performed.

The brightness value comparison unit 52 makes it possible to detect periodicity not seen in the first embodiment by observing the time change of the brightness average value from a relative viewpoint. In addition to the calculation of the luminance difference of the first embodiment, the luminance difference calculation units 60A and 60B calculate the luminance difference between the luminance average value 310A and the luminance average value 310B at the same time. The method for determining the periodicity based on the brightness difference may be the same as in the first embodiment.

In FIG. 19, at times t1 to t4, the difference in luminance difference between the average luminance value 310A and the average luminance value 310B is equal to or greater than the threshold value A.

<Operation procedure>
FIG. 20 is an example of a flowchart showing a procedure in which the image pickup apparatus 5 switches from a 360-degree image to a 180-degree image. In the description of FIG. 20, the difference from FIG. 14 will be mainly described.

First, in step S12-2, the brightness difference calculation units 60A and 60B calculate the brightness difference between the brightness average value of the frame to be processed and the brightness average value of the previous frame, and the difference in brightness difference in the brightness value feature storage unit 61. I (an example of the difference in the first luminance difference) is stored (comparison in the same input image processing modules 34A and 34B). Further, the luminance difference calculation unit 60A calculates the luminance difference II (an example of the second luminance difference difference) from the luminance average value output by the input image processing module 34B at the same time as the frame to be processed, and the luminance difference. The calculation unit 60B calculates the difference II of the brightness difference from the brightness average value output by the input image processing module 34A at the same time as the processing target frame (comparison between the input image processing modules 34A and 34B).

Then, in steps S13 to S15, the difference I in the brightness difference and the difference II in the brightness difference are similarly compared with the threshold values A and B (S13 to S15).

Then, when the comparison with the threshold value A and the threshold value B satisfies the criteria for either the difference I in the brightness difference or the difference II in the brightness difference, the brightness value comparison unit 52 detects the gesture switching operation (S16). Subsequent processing may be the same as in Example 1.

<Main effect>
According to this embodiment, in addition to the effect of the first embodiment, the gesture switching operation is detected even when the periodic change of the average luminance value does not appear remarkably even though the user is performing the gesture switching operation. The accuracy can be improved.

In this embodiment, the image pickup device 5 capable of detecting the gesture switching operation even when the gesture position of the user is off the center of the lens will be described. In this embodiment, the periodicity of the time change of the average brightness value is determined for each area in which the image data is divided. For example, when the image data is divided into four, the image pickup apparatus 5 has four luminance value storage units 51 and four luminance value comparison units 52.

<Division example>
FIG. 21 is a diagram showing an example of dividing image data. In FIG. 21, the image data is divided into four areas 320 by a horizontal line and a vertical line passing through the center. It may be divided at an equal angle from the center, or it may be divided into different shapes for each area. Further, it is sufficient that there are two or more areas 320, and four are only examples.

<About functions>
FIG. 22 is a diagram showing a configuration example of the input image processing modules 34A and 34B of this embodiment. Since the input image processing modules 34A and 34B divide the image data and perform processing, the input image processing modules 34A and 34B have the number of the luminance value storage unit 51 and the luminance value comparison unit 52 divided. Further, it has a luminance area dividing unit 56 that divides image data into each area, and an effective luminance area selecting unit 57 that selects the comparison result of the area in which the gesture switching operation is detected and outputs it to the distortion correction / image composition module 36. ..

The function of the luminance value comparison unit 52 may be the same as that of FIG. 11B of the first embodiment.

<About operation>
FIG. 23 is an example of a flowchart for explaining the processing of the input image processing module of this embodiment.

First, the luminance area dividing unit 56 receives the input image (image data) for which gamma correction and the like have been completed, and divides the input image (image data) into a set number of areas (S101). The number of areas may be arbitrary, but may be, for example, two or more. Further, the shape of the area may be a rectangle similar to the original image data, a shape divided at equal intervals from the center, or an indeterminate shape. The area of each area does not have to be the same.

Next, the exposure control unit 50 calculates the brightness average value for each divided area (S102).

Next, the exposure control unit 50 transfers each brightness average value to a plurality of brightness value storage units 51 corresponding to the area (S103).

The brightness value comparison unit 52 detects the gesture switching operation for each area in the same manner as in the first embodiment (S104). That is, the brightness value comparison unit 52 in each area executes steps S12 to S15 of FIG. Since the gesture switching motion is detected for each area, it becomes easy to detect the gesture switching motion even if the user's hand is away from the center of the lens.

Each luminance value comparison unit 52 transfers the comparison result for each area to the effective luminance area selection unit 57 (S105). This comparison result is the presence or absence of the detection result of the gesture switching operation associated with the area identification information.

If there is even one area in which the gesture switching operation is detected (S106), the effective luminance area selection unit 57 determines that the user has performed the gesture operation (S107). Subsequent processing may be the same as in FIG. 14 of Example 1.

<Main effect>
According to this embodiment, in addition to the effect of the first embodiment, the gesture switching operation can be detected even when the gesture position of the user is off the center of the lens.

In this embodiment, an imaging device 5 capable of detecting a gesture switching operation different from that of the first to fourth embodiments will be described. For example, the image pickup device 5 can detect the gesture switching operation even when the user holds his / her hand over the lens in a circular motion.

<About functions>
FIG. 24 is an example of a diagram showing the functions of the input image processing modules 34A and 34B of this embodiment. In this embodiment, since the image data is divided and processed, the input image processing modules 34A and 34B have the luminance value storage unit 51 and the luminance value comparison unit 52 for the number of the divided images. Further, the brightness area dividing unit 56 for dividing the image data into the area for obtaining the average value of the brightness and the gesture is detected from the change in the brightness difference for each frame of the plurality of brightness value comparison units 52 to correct the distortion and synthesize the image. It has a gesture detection unit 58 that outputs to the module 36. The gesture detection unit 58 acquires the change in the brightness difference for each frame of the brightness value comparison unit 52 from each brightness value comparison unit 52, and detects the gesture switching operation from the change in the brightness difference in each area.

The function of the luminance value comparison unit 52 may be the same as that of FIG. 11B of the first embodiment.

<Gesture switching operation and change in brightness difference in each area>
FIG. 25 is a diagram schematically showing a change in brightness difference in each area of this embodiment. In FIG. 25, the original image data will be described as being divided into four parts. Also, the user is moving his hand in a clockwise circle on the lens. When divided into four by a straight line passing through the center of the image data, FIG. 25 (a) is the brightness average value of the upper left image data, and FIG. 25 (b) is the brightness average value of the upper right image data. c) is the brightness average value of the lower right image data, and FIG. 25 (d) is the brightness average value of the lower left image data. Since it is clockwise, the user's hand moves in the order of upper left → upper right → lower right → lower left.

As can be seen from FIG. 25, the portion where the average luminance value is small (downward peak) moves to the later side in the order of upper left → upper right → lower right → lower left. That is, when comparing the average brightness values of each area, it can be seen that there is periodicity. When the gesture detection unit 58 detects the periodicity shown in FIG. 25, it determines that the gesture switching operation has been detected.

The luminance value feature determination unit 62 corresponding to each area determines the midpoint P between the two frames in which the difference in luminance difference is equal to or greater than the threshold value A as the frame having the smallest luminance. The midpoint P is conditioned on the condition that the difference between the two timings that are equal to or higher than the threshold value A is less than the threshold value B. Then, the gesture detection unit 58 confirms the periodicity based on the frame (midpoint P of each area) where the brightness of the four areas is the smallest.

FIG. 25 (e) shows the intervals of frames f1 to f5 (time t1 to t5 may be used) in which the brightness is the smallest in each area. The gesture detection unit 58 has a difference between the distance between f1 and f2 and a distance between f2 and f3, a difference between the distance between f2 and f3 and a difference between f3 and f4, and a difference between the distance between f3 and f4 and the distance between f4 and f5. If it is less than the threshold C (an example of the third threshold), it is determined that the average brightness of the area has periodicity, and the gesture switching operation is detected.

<Operation procedure>
FIG. 26 is an example of a flowchart for explaining the processing of the input image processing module of this embodiment.

First, the luminance area dividing unit 56 divides the input image (original image data) into areas (S201).

Next, the exposure control unit 50 calculates the brightness average value for each divided area (S202).

Next, the exposure control unit 50 stores the respective brightness average values in the plurality of brightness value storage units 51 corresponding to the areas (S203).

Next, the brightness value comparison unit 52 corresponding to each area determines a frame in which the brightness average value takes the minimum value with respect to the brightness average value stored in the brightness value storage unit 51 in the same manner as in the first embodiment (the same method as in the first embodiment). S204). That is, the brightness value comparison unit 52 in each area executes steps S12 to S15 in FIG. 14, and is a midpoint between two timings in which the difference in brightness is equal to or greater than the threshold value A and the difference in timing is less than the threshold value B. Determine P (or the frame or time closest to the midpoint P).

Next, each luminance value comparison unit 52 transfers all the frames having the minimum luminance average value to the gesture detection unit 58 (S205).

Next, the gesture detection unit 58 aggregates the frames (timing) in which the average brightness value of each area takes the minimum value, and makes a gesture when the difference between the frames taking the minimum value in each area is less than the preset threshold value C. It is determined that the operation has been detected (S206).

<Main effect>
According to this embodiment, in addition to the effect of the first embodiment, the image pickup device 5 can detect the gesture switching operation even when the user holds his / her hand over the lens in a circular motion.

In this embodiment, the image pickup device 5 that can improve the accuracy when the gesture position of the user is off the center of the lens and the time change of the average luminance value does not appear remarkably will be described.

<About functions>
First, it is assumed that the configurations of the input image processing modules 34A and 34B are the same as those in FIG. 22 of the fourth embodiment. Therefore, in this embodiment as well, the image data of each of the two lenses is divided in the same manner.

FIG. 27 is a diagram showing a configuration example of the brightness value comparison unit 52 of this embodiment. The configuration of the luminance value comparison unit 52 is the same as that of FIG. 18 of the third embodiment. That is, the brightness average value output by the other input image processing modules 34A and 34B is also input to the brightness difference calculation units 60A and 60B. Further, the input luminance average value is calculated for each area. Therefore, the brightness difference calculation units 60A and 60B can calculate not only the brightness difference for each area in the own lens but also the brightness difference between the brightness average value of the other lens and the brightness average value of the own lens for each area. By calculating the relative luminance difference, it becomes possible to detect the feature of the luminance difference in more detail.

When one image data is divided into four, there are 4 × 4 = 16 comparison combinations between the input image processing modules 34A and 34B, but in this embodiment, the areas located in point symmetry of both lenses are separated from each other. All shall be compared.

FIG. 28 is an example of a diagram illustrating an object of comparison. FIG. 28A is the image data A of the input image processing module 34A, and FIG. 28B is the image data B of the input image processing module 34B. Therefore, the luminance value feature determination unit 62 compares the luminance average values with the following combinations.
-Area 3 of image data B is point-symmetrical to area 1 of image data A.
-Area 4 of image data B is point-symmetrical to area 2 of image data A.
-Area 1 of image data B is point-symmetrical with area 3 of image data A.
-Area 2 of image data B is point-symmetrical to area 4 of image data A.
By doing so, it is possible to compare the brightness average value of the image data area affected by the hand position and the image data area less affected by the hand position. Areas having a line-symmetrical relationship may be compared with each other.

The symmetry of the comparison in FIG. 28 is only an example, and any of the areas 1 of the image data A and the areas 1 to 4 of the image data B, and any of the areas 2 of the image data A and the areas 1 to 4 of the image data B. Alternatively, either the area 3 of the image data A and the areas 1 to 4 of the image data B, or any of the areas 4 of the image data A and the areas 1 to 4 of the image data B may be compared.

<Gesture switching operation and change in brightness difference in each area>
FIG. 29 is a diagram showing an example of a time change of the average luminance value due to the gesture switching operation. FIG. 29 (a) is, for example, the brightness average value of the upper left area of the input image processing module 34A, FIG. 29 (b) is the brightness average value of the upper right area, and FIG. 29 (c) is the brightness average value of the lower right area. It is the brightness average value, and FIG. 29 (d) is the brightness average value of the lower left area. The average luminance value of each area is indicated by “◯”, and the average luminance value of the input image processing module 34B to be compared is indicated by “Δ”.

The luminance value feature determination units 62A and 62B identify a frame in which the difference in luminance difference from the average luminance value of the other lens is equal to or greater than the threshold value A, and when this frame is detected at intervals less than the threshold value B, a gesture switching operation is performed. Is detected. In FIG. 29 (a), the periodicity is detected by the average luminance value in the upper left area. The brightness value feature determination units 62A and 62B corresponding to each area output to the effective brightness area selection unit 57 that the gesture switching operation has been detected. The effective luminance area selection unit 57 determines that the user has performed the gesture operation if there is even one area in which the gesture switching operation is detected.

<Operation procedure>
FIG. 30 is an example of a flowchart for explaining the processing of the input image processing module of this embodiment. In the description of FIG. 30, the difference from FIG. 23 will be mainly described.

In FIG. 30, in step S103-2, the exposure control unit 50 transfers the respective luminance average values to the plurality of luminance value storage units 51 corresponding to the areas, and the luminance average value of the area at the point-symmetrical position of the other lens. Also forward.

Next, in step S104-2, the brightness value comparison unit 52 corresponding to each area compares the brightness average value stored in the brightness value storage unit 51 with the brightness average value of the other lens, and calculates the brightness difference. Then, the gesture switching operation is detected as in the first embodiment. Subsequent processing may be the same as in FIG.

<Main effect>
According to this embodiment, in addition to the effect of the first embodiment, it is possible to improve the accuracy when the gesture position of the user is off the center of the lens and when the time change of the average luminance value does not appear remarkably. ..

In this embodiment, an image pickup device 5 capable of detecting a three-dimensional gesture switching operation will be described.

<About functions>
FIG. 31 is a diagram illustrating the configuration of the input image processing modules 34A and 34B of this embodiment. The configurations of the input image processing modules 34A and 34B are the same as those in FIG. 24 of the fifth embodiment, but the gesture detection units 58A and 58B (examples of the first detection unit and the second detection unit, respectively) are the input image processing modules 34A. , 34B, the data received from the respective luminance value comparison units 52 is transferred to the other (input image processing modules 34B, 34A). Since the same data (periodic data) is input to each gesture detection unit 58, the gesture can be recognized by either the input image processing modules 34A and 34B.

<Operation procedure>
FIG. 32 is an example of a flowchart for explaining the processing of the input image processing module of this embodiment.

First, as described in the fifth embodiment, each luminance value comparison unit 52 determines a frame in which the average luminance value takes a minimum value, and transfers all of the frames to the gesture detection unit 58 (S301).

Next, the gesture detection unit 58A transfers the frame having the minimum brightness average value to the gesture detection unit 58B of the other input image processing module 34B (S302). The gesture detection unit 58B transfers a frame having a minimum brightness average value to the gesture detection unit 58A of another input image processing module 34A.

The gesture detection units 58A and 58B aggregate the frames (timing) in which the brightness difference in each area becomes large, and detect the gesture operation when the difference between the frames that take the minimum value in each area is less than the preset threshold value C. (S303). That is, the gesture detection units 58A and 58B determine whether or not there is a gesture operation for each of the frames in which the average brightness value of the input image processing module and the other input image processing modules takes the minimum value.

<Main effect>
According to this embodiment, in addition to the effect of the first embodiment, a three-dimensional gesture switching operation can be detected.

<Other application examples>
Although the best mode for carrying out the present invention has been described above with reference to examples, the present invention is not limited to these examples, and various modifications are made without departing from the gist of the present invention. And substitutions can be made.

For example, in the present embodiment, the inside view of real estate has been described as an example, but the application example of the image communication system 100 is not limited to this. For example, it can be applied when pointing to a local object such as an exhibition, an exhibition, a factory tour, sightseeing, or a tour.

Further, in the present embodiment, a human image is taken by the image pickup device 5, but a machine, a robot, or an animal may point to the object.

Further, the configuration example shown in FIG. 11 and the like is divided according to the main functions in order to facilitate understanding of the processing by the image communication system 100. The present invention is not limited by the method of dividing the processing unit or the name. The processing of the image communication system 100 can be divided into more processing units according to the processing content. It is also possible to divide one processing unit so as to include more processing.

A plurality of communication management systems 80 may exist, or the functions of the communication management system 80 may be distributed to a plurality of servers. There may be a relay device that relays image data and sound data.

Each function of the embodiment described above can be realized by one or more processing circuits. Here, the "processing circuit" in the present specification is a processor programmed to execute each function by software such as a processor implemented by an electronic circuit, or a processor designed to execute each function described above. It shall include devices such as ASIC (Application Specific Integrated Circuit), DSP (digital signal processor), FPGA (field programmable gate array) and conventional circuit modules.

5 Image sensor 10 Terminal device 21 Image sensor 34 Input image processing module 50 Exposure control unit 51 Luminance value storage unit 52 Luminance value comparison unit 70 Communication management system 100 Image communication system

JP-A-2019-012881

Claims (11)

An image pickup device that synthesizes image data captured by a plurality of image sensors to generate image data having a wider angle than the image data captured by one image sensor.
When the average brightness of the image data captured by one of the image sensors has periodicity,
An image pickup apparatus characterized in that the image data having a periodic luminance average value is not used for synthesis, but the image data captured by the other one or more image pickup elements is output. A brightness value storage unit that stores the average brightness of the image data of each frame, and a brightness value storage unit.
A brightness difference calculation unit that calculates the brightness difference between the brightness average value of the frame and the brightness average value of the previous frame stored in the brightness value storage unit, and the difference between two temporally adjacent brightness differences.
It has a luminance value feature determination unit for determining whether or not the absolute value of the difference in luminance difference is equal to or greater than the first threshold value.
In the brightness value feature determination unit, the interval between the frame in which the difference in brightness difference equal to or greater than the first threshold value is detected and the frame in which the difference in brightness difference equal to or greater than the first threshold value is detected last time is set. The imaging device according to claim 1, wherein when the value is less than the second threshold value, it is determined that the luminance average value has periodicity. For one cycle of frames for which periodicity detection has been completed, the difference in brightness difference equal to or greater than the first threshold value and the time when the difference in brightness is detected are stored in the brightness value storage unit. The imaging device according to claim 2, further comprising a feature compression unit that deletes frames for one cycle already stored in the luminance value storage unit. A first luminance difference calculation unit that calculates the first luminance difference between the first image data captured by the first image sensor and the second image data captured by the second image sensor.
A second luminance difference calculation unit that calculates a second luminance difference between the second image data captured by the second image sensor and the first image data captured by the first image sensor. ,
The frame in which the difference in the first luminance difference between the two temporally adjacent first luminance differences and equal to or greater than the first threshold value is detected, and the frame in which the difference in the first luminance difference equal to or greater than the first threshold value is detected, and the previous time, the said When the distance from the frame in which the difference in the first luminance difference is detected is less than the second threshold value, the first luminance value feature determination unit that determines that the average luminance value has periodicity, and the first luminance value feature determination unit.
The frame in which the difference between the two temporally adjacent first luminance differences and the difference in the second luminance difference equal to or greater than the first threshold value is detected, and the previous time, the first luminance difference equal to or greater than the first threshold value. When the distance from the frame in which the difference between the two luminance differences is detected is less than the second threshold value, the second luminance value feature determination unit that determines that the luminance average value has periodicity, and the second luminance value feature determination unit.
The imaging device according to claim 2 or 3, wherein the image pickup apparatus is characterized by having. It has an area division unit that divides the image data into a plurality of areas.
The luminance value feature determination unit determines whether or not the luminance average value has periodicity for each area, and determines whether or not the luminance average value has periodicity.
The imaging device according to claim 2 or 3, wherein when even one of the areas having periodicity is detected, it is determined that the brightness average value of the image data has periodicity. It has an area division unit that divides the image data into a plurality of areas.
The brightness difference calculation unit has, for each area, the brightness average value of the frame, the brightness average value of the previous frame of the frame, the brightness difference of the brightness average value of the frame and the previous frame, and two temporally adjacent ones. Calculate the difference in brightness and
The luminance value feature determining unit detected the difference between the frame in which the difference in luminance difference equal to or greater than the first threshold value was detected and the difference in luminance difference equal to or greater than the first threshold value in the previous time for each area. If the distance from the frame is less than the second threshold, determine the time of the midpoint between the two frames and
The imaging according to claim 2 or 3, further comprising a detection unit for determining that the luminance average value has periodicity when the time difference in each area is less than a preset third threshold value. apparatus. An area division unit that divides the image data into a plurality of areas,
The first luminance difference between the predetermined area of the first image data captured by the first image sensor and the predetermined area of the second image data captured by the second image sensor, and the temporal period. The first luminance difference calculation unit that calculates the difference between the two first luminance differences adjacent to the
A second luminance difference between a predetermined area of the second image data captured by the second image sensor and a predetermined area of the first image data captured by the first image sensor, and A second luminance difference calculation unit that calculates the difference between two second luminance differences that are adjacent in time,
The interval between the frame in which the difference in the first luminance difference equal to or greater than the first threshold value is detected and the frame in which the difference in the first luminance difference equal to or greater than the first threshold value is detected last time is the second. If it is less than the threshold value of, the first luminance value feature determination unit that determines that the average luminance value has periodicity, and
The interval between the frame in which the difference in the second luminance difference above the first threshold value is detected and the frame in which the difference in the second luminance difference above the first threshold value is detected last time is the second. If it is less than the threshold value of, it has a second luminance value feature determination unit having periodicity in the luminance average value, and has.
When the first luminance value feature determination unit or the second luminance value feature determination unit determines that there is at least one of the areas having periodicity in the average luminance value, the average luminance value of the image data has periodicity. The imaging apparatus according to claim 2 or 3, wherein it is determined that there is. For each of the areas, the first is to calculate the brightness difference between the brightness average value of the frame imaged by the first image sensor and the brightness average value of the previous frame, and the difference between the two time-adjacent brightness differences. Luminance difference calculation unit and
For each of the areas, the interval between the frame in which the difference in luminance difference equal to or greater than the first threshold value is detected and the frame in which the difference in luminance difference equal to or greater than the first threshold value is detected last time is second. If it is less than the threshold value, the first luminance value feature determination unit that determines the time of the midpoint between the two frames, and
When the time difference in each area is less than a preset third threshold value, the first detection unit that determines that the luminance average value has periodicity, and
For each of the areas, the brightness difference between the brightness average value of the frame imaged by the second image sensor and the brightness average value of the previous frame, and the difference between the two time-adjacent brightness differences are calculated. Luminance difference calculation unit and
For each of the areas, the interval between the frame in which the difference in luminance difference equal to or greater than the first threshold value is detected and the frame in which the difference in luminance difference equal to or greater than the first threshold value is detected last time is second. If it is less than the threshold value, a second luminance value feature determination unit that determines the time of the midpoint between the two frames, and
It has a second detection unit that determines that the luminance average value has periodicity when the time difference in each area is less than a preset third threshold value.
The imaging device according to claim 6, wherein the first detection unit and the second detection unit transmit the difference in time in each area to each other. The image pickup apparatus according to any one of claims 1 to 8, wherein the image pickup device has three or more image pickup elements. This is an image generation method in which an image pickup device generates image data having a wider angle than the image data captured by one image sensor by synthesizing the image data captured by a plurality of image sensors.
When the average brightness of the image data captured by one of the image sensors has periodicity,
An image generation method characterized in that the image data having periodicity is not used for synthesis and the image data captured by the other image sensor is output. For an image pickup device that synthesizes image data captured by a plurality of image sensors to generate image data having a wider angle than the image data captured by one image sensor.
When the average brightness of the image data captured by one of the image sensors has periodicity,
A program characterized in that the image data having periodicity is not used for synthesis, but the image data captured by the other image sensor is output.

Images