JP2024003395A - Photography device, image processing method, communication system, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce work of setting an area of interest in a wide-field image such as a panoramic image in a fixed display area 440 by solving a problem in which, in a conventional technology, a participant (user) at a video distribution source sets the area of interest in a panoramic image in the fixed display area 440, which made it difficult to prepare for video distribution such as a video conference.

SOLUTION: A photography device 6 automatically detects a rectangular display 280 of a display device 2, which is mainly treated as an area of interest to enlarge the area of interest within a wide-field image and set the area of interest as a fixed display area 440. This has an effect in which work of a user of a video distribution source can be reduced.

SELECTED DRAWING: Figure 13

COPYRIGHT: (C)2024,JPO&INPIT

Description

The present disclosure relates to a photographing device, an image processing method, a communication system, and a program.

A wide-angle (fish-eye) image taken using an ultra-wide-angle (fish-eye) lens is developed into a rectangular panoramic image (360° panoramic image if it is a fish-eye) in a shooting device used for video distribution such as video conferencing. The technology is known. In the case of a video conference, by creating a panoramic image, the participants at the video distribution destination can simultaneously understand all the participants at the video distribution source (see display area 420 in FIG. 13).

Furthermore, since a panoramic image is an elongated image, conventional devices enlarge an image of a region of interest within the panoramic image and fit it into a fixed display area of the screen (see fixed display area 440 in FIG. 13). It is also possible to generate a composite image and distribute it to a video distribution destination (see Non-Patent Document 1). This makes it easier for participants at the video distribution destination to grasp images of interest, such as materials.

However, with conventional technology, participants (users) at the video distribution source had to set the noteworthy area in the panoramic image as a fixed display area, so they had to prepare for video distribution such as a video conference. There was a problem that it took a lot of time.

The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to reduce the work of setting a noteworthy area in a wide-field image such as a panoramic image as a fixed display area.

The invention according to claim 1 is a photographing device that outputs an image for video distribution, and includes: a wide-field image generation unit that generates a wide-field image from a fisheye image obtained by photographing; a code information detection unit that detects included code information; and an instruction information detection unit that detects instruction information to detect a rectangular area from the code information when the code information is detected by the code information detection unit. a rectangular area detection unit that detects a predetermined rectangular area outside the code information in the wide-field image when the instruction information is detected by the instruction information detection unit; and the rectangular area detection unit If the predetermined rectangular area is detected by, a composite image generation unit that generates a predetermined composite image by combining a predetermined rectangular area image indicating the predetermined rectangular area in the wide-field image and the wide-field image. and a transmitter that transmits the predetermined composite image generated by the composite image generator to the outside.

As explained above, according to the present invention, it is possible to reduce the work of setting a noteworthy area in a wide-field image as a fixed display area.

1 is an overall configuration diagram of a communication system according to an embodiment. It is an image diagram of the use of the photographing device at an arbitrary base. (a) is a plan view of a fisheye lens, and (b) is a conceptual diagram of a fisheye image. FIG. 3 is a diagram showing a photographing target range (in the vertical direction) of the photographing device. FIG. 2 is an electrical hardware configuration diagram of a display device. FIG. 2 is an electrical hardware configuration diagram of a communication terminal and a server. FIG. 2 is an electrical hardware configuration diagram of the photographing device. FIG. 2 is a functional configuration diagram of a display device, a communication terminal, and a photographing device. 7 is a flowchart illustrating a process of specifying a fixed display area by the photographing device. It is a flowchart which shows the process of transmitting predetermined content information from a photographing device to a communication terminal. FIG. 2 is a diagram showing a two-dimensional code displayed on a display of a display device. FIG. 2 is a diagram showing a two-dimensional code displayed on a display of a display device. It is a diagram showing a screen displayed by the communication terminal 5 at the base α. FIG. 2 is a diagram showing an example of a two-dimensional code displayed by a display device. It is a figure which shows an example of the display method of the two-dimensional code displayed by a display apparatus. It is a figure which shows an example of the display method of the two-dimensional code displayed by a display apparatus. It is a figure which shows an example of the display method of the two-dimensional code displayed by a display apparatus. It is a figure which shows an example of the display method of the two-dimensional code displayed by a display apparatus. It is a figure which shows an example of the display method of the two-dimensional code displayed by a display apparatus. It is a figure which shows an example of the display method of the two-dimensional code displayed by a display apparatus. FIG. 7 is a diagram illustrating an example of a method of displaying a two-dimensional code displayed by a display device when the camera included in the photographing device is a hemispherical camera. It is a figure which shows an example of the display method of the barcode displayed by a display apparatus.

Embodiments of the present invention will be described below with reference to the drawings.

〔overall structure〕
FIG. 1 is an overall configuration diagram of a communication system according to an embodiment.

As shown in FIG. 1, the communication system 1 of this embodiment includes a display device 2, a communication terminal 5, and a photographing device 6 installed at a first base (base α), and a communication control server 7 on the cloud. , is constructed by the communication terminal 10 installed at the second base (base β). Further, the communication terminal 5 is a PC (Personal Computer), a video conference terminal, or the like, and is capable of communicating with the communication control server 7 via a communication network 100 such as the Internet. Similarly, the communication terminal 10 is a PC, a video conference terminal, or the like, and can communicate with the communication control server 7 via the communication network 100. Thereby, the communication terminal 5 and the communication terminal 10 can communicate via the communication control server.

Note that the display device 2 and the photographing device 6 may be provided on the second base β side as well as on the first base α side.

The display device 2 is an electronic whiteboard, but may be a simple display.

Further, although two bases are shown in FIG. 1, it is also possible to communicate with three or more bases. In this case, the number of communication terminals increases according to the number of bases.

The photographing device 6 transmits to the communication terminal 5 video data obtained by photographing the surroundings and sound data obtained by collecting surrounding sounds. This transmission method may be wired or wireless.

Further, at the base α, the communication terminal 5 acquires the video data and sound data output from the photographing device 6, and transmits them to the communication control server 7, and the communication control server 7 receives the communication terminal at the base β, which is the other party. The video data and sound data are transferred to 10. Similarly, at the base β, the communication terminal 10 acquires video data and sound data from the surrounding area and transmits the data to the communication control server 7, and the communication control server 7 sends the video data to the communication terminal 5 at the base α, which is the other party. and transfer sound data.

[Usage image]
FIG. 2 is an image diagram of how the photographing device is used at an arbitrary base.

As shown in FIG. 2, for example, four people participate in a video distribution such as a meeting at base α, and a photographing device 6 is installed on a desk 110. The photographing device 6 is provided with a fisheye lens FL, which will be described later, and is capable of photographing the surrounding 360°. The display device 2 displays materials, etc. (shared screen). In this case, the communication terminal 5 may be a PC of one of the participants (for example, the organizer) shown in FIG. 2.

[Shooting range of the photographing device]
Next, the photographing target range of the photographing device 6 will be explained using FIGS. 3 and 4. In FIG. 3, (a) is a plan view of a fisheye lens, and (b) is a conceptual diagram of a fisheye image. FIG. 4 is a diagram showing the photographing target range (in the vertical direction) of the photographing device.

As shown in FIG. 3(a), the lens of the photographing device 6 is provided with a fisheye lens FL having an angle of view of 180° or more. Further, as shown in FIG. 3(b), the center of the fisheye image (video) is set as an invalid area, and the donut area is set as a valid area.

The photographing target range of the donut region is as shown in FIG. 4 when viewed from the side. The image toward the ceiling is cut, and the image toward the floor is removed from the imaging range. Even so, since the area where the person is present can be covered, it is possible to photograph the head (or face) and chest of a participant who is having a meeting using a desk.

[Hardware configuration]
Next, the electrical hardware configurations of the display device 2, communication terminals 5 and 10, photographing device 6, and communication control server 7 will be described using FIGS. 5 to 7.

<Hardware configuration of display device>
First, the electrical hardware configuration of the display device 2 will be described using FIG. 5. FIG. 5 is an electrical hardware configuration diagram of the display device.

As shown in FIG. 5, FIG. 2 is a hardware configuration diagram of the display device. As shown in FIG. 2, the display device 2 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, an SSD (Solid State Drive) 204, a network I/ F205 and an external device connection I/F (Interface) 206.

Among these, the CPU 201 controls the operation of the display device 2 as a whole. The ROM 202 stores programs used to drive the CPU 201, such as the CPU 201 and an IPL (Initial Program Loader). RAM 203 is used as a work area for CPU 201. The SSD 204 stores various data such as programs for display devices. Network controller 205 controls communication with communication network 100 . External device connection I/F 206 is an interface for connecting various external devices. The external devices in this case are, for example, a USB (Universal Serial Bus) memory 230 and external devices (microphone 240, speaker 250, camera 260).

The display device 2 also includes a capture device 211, a GPU 212, a display controller 213, a contact sensor 214, a sensor controller 215, an electronic pen controller 216, a short-range communication circuit 219, an antenna 219a of the short-range communication circuit 219, and a power switch 222 selected. It includes switches 223 and a display 280. Note that the display 280 is an example of a display section. The display unit includes a screen projected by a projector (including a display target of projection mapping).

Among these, the capture device 211 causes the display 280 to display video information as a still image or a moving image. A GPU (Graphics Processing Unit) 212 is a semiconductor chip that specializes in graphics. The display controller 213 controls and manages screen display in order to output the output image from the GPU 212 to the display 280 or the like. The contact sensor 214 detects when the electronic pen 290, the user's hand H, or the like comes into contact with the display 280. Sensor controller 215 controls processing of contact sensor 214. The contact sensor 214 performs coordinate input and coordinate detection using an infrared cutoff method. This method of inputting and detecting coordinates is such that two light receiving and emitting devices installed at both ends of the upper side of the display 280 emit a plurality of infrared rays in parallel to the display 280. This is a method of receiving light that is reflected by a reflecting member and returns along the same optical path as the light emitted by the light receiving element. The contact sensor 214 outputs the ID of the infrared rays emitted by the two light receiving and emitting devices blocked by the object to the sensor controller 215, and the sensor controller 215 identifies the coordinate position that is the contact position of the object. The electronic pen controller 216 communicates with the electronic pen 290 to determine whether or not the display 280 is touched with the tip of the pen or the end of the pen. The near field communication circuit 219 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The power switch 222 is a switch for switching the power of the display device 2 on and off. The selection switches 223 are, for example, a group of switches for adjusting the brightness, shade, etc. of the display on the display 280.

Furthermore, the display device 2 includes a bus line 210. The bus line 210 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 201 shown in FIG. 2.

Note that the contact sensor 214 is not limited to the infrared cutoff type, but can also be a capacitive type touch panel that identifies the touch position by detecting changes in capacitance, or a touch sensor that identifies the touch position by voltage changes between two opposing resistive films. Various detection means may be used, such as a resistive touch panel that uses a resistive film, or an electromagnetic induction touch panel that detects electromagnetic induction caused by a contact object coming into contact with the display section to specify the contact position. Furthermore, the electronic pen controller 216 may determine whether or not a portion of the electronic pen 290 that the user grasps or other portions of the electronic pen is touched, in addition to the pen tip and pen tail of the electronic pen 290.

<Hardware configuration of communication terminal and communication control server>
Next, the electrical hardware configuration of the communication terminal 5 will be described using FIG. 6. FIG. 5 is an electrical hardware configuration diagram of the display device. As shown in FIG. 5, the communication terminal 5 is constructed by a computer, and includes a CPU 501, ROM 502, RAM 503, HD (Hard Disk) 504, HDD (Hard Disk Drive) controller 505, display 506, external device connection I /F (Interface) 508, network I/F 509, bus line 510, keyboard 511, pointing device 512, optical drive 514, and media I/F 516.

Among these, the CPU 501 controls the operation of the communication terminal 5 as a whole. The ROM 502 stores programs used to drive the CPU 501 such as IPL. RAM 503 is used as a work area for CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls reading and writing of various data to the HD 504 under the control of the CPU 501. The display 506 displays various information such as a cursor, menu, window, characters, or images. External device connection I/F 508 is an interface for connecting various external devices. The external devices in this case include, for example, the display device 2, a display device, a USB (Universal Serial Bus) memory, and a printer. The network I/F 509 is an interface for data communication using a network. The bus line 510 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 501 shown in FIG. 6.

Further, the keyboard 511 is a type of input means that includes a plurality of keys used for inputting characters, numbers, various instructions, and the like. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. The optical drive 514 controls reading and writing of various data to and from an optical storage medium 513, which is an example of a removable recording medium. Note that the optical storage medium 513 may be a CD, DVD, Blu-ray (registered trademark), or the like. The media I/F 516 controls reading or writing (storage) of data to a recording medium 515 such as a flash memory.

Note that the communication control server 7 and the communication terminal 10 have the same configuration as the communication terminal 5, so a description thereof will be omitted.

<Hardware configuration of imaging device>
Next, the electrical hardware configuration of the imaging device will be described using FIG. 7. FIG. 7 is an electrical hardware configuration diagram of the imaging device.

As shown in FIG. 7, in the following, the photographing device 6 is a device that uses an image sensor to image a 360° surrounding video at a predetermined height. Any number is fine. Further, it does not necessarily have to be a dedicated device, and a 360° video imaging unit may be attached to a PC, digital camera, smartphone, etc. afterward to have substantially the same functions.

As shown in FIG. 7, the imaging device 6 includes an imaging unit 601, an image processing unit 604, an imaging control unit 605, a microphone 608, a sound processing unit 609, a CPU (Central Processing Unit) 611, and a ROM (Read Only Memory). ) 612, SRAM (Static Random Access Memory) 613, DRAM (Dynamic Random Access Memory) 614, operation unit 615, external device connection I/F 616, communication unit 617, antenna 617a, audio sensor 618, and speaker 619. ing.

Among these, the imaging unit 601 includes a wide-angle lens (so-called fisheye lens) 602 having an angle of view of 360° for forming a hemispherical image, and an imaging element 603 (image sensor) provided corresponding to each wide-angle lens. It is equipped with The image sensor 603 is an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts an optical image formed by the fisheye lens 602 into electrical signal image data and outputs the image data, and a horizontal or vertical It includes a timing generation circuit that generates synchronization signals and pixel clocks, and a group of registers in which various commands and parameters necessary for the operation of this image sensor are set. Note that the imaging unit 601 may be a 360° camera, and is an example of an imaging means capable of imaging the surrounding area of the imaging device 6 at 360°.

The imaging elements 603 (image sensors) of the imaging unit 601 are each connected to an image processing unit 604 via a parallel I/F bus. On the other hand, the imaging element 603 of the imaging unit 601 is connected to the imaging control unit 605 via a serial I/F bus (such as an I2C bus). The image processing unit 604, the imaging control unit 605, and the sound processing unit 609 are connected to the CPU 611 via a bus 610. Furthermore, a ROM 612, an SRAM 613, a DRAM 614, an operation section 615, an external device connection I/F 616, a communication section 617, an audio sensor 618, a speaker 619, and the like are also connected to the bus 610.

The image processing unit 604 takes in the image data output from the image sensor 603 through the parallel I/F bus, performs predetermined processing on the image data, and converts the fisheye image into wide-field image and speaker image data. create. Note that the "wide-field image" is an image with a wider field of view than a general image obtained by photographing with a single-lens reflex camera or the like, and is an image with a viewing angle of 180 degrees or more. Wide-field images include omnidirectional images, hemispherical images, 3D panoramic images, 2D panoramic images, VR images, and the like.

Further, the image processing unit 604 combines the wide-field image, the speaker image, etc., and outputs one moving image.

Generally, the imaging control unit 605 sets commands and the like in a register group of the imaging device 603 using the I2C bus, with the imaging control unit 605 as a master device and the imaging device 603 as a slave device. Necessary commands and the like are received from the CPU 611. The imaging control unit 605 also uses the I2C bus to take in status data and the like of the register group of the imaging device 603 and sends it to the CPU 611.

Further, the imaging control unit 605 instructs the imaging device 603 to output image data at the timing when the imaging start button on the operation unit 615 is pressed or at the timing when an imaging start instruction is received from the PC. Depending on the photographing device 6, it may have a preview display function or a function corresponding to video display on a display (for example, a display of a PC or a smartphone). In this case, image data is continuously output from the image sensor 603 at a predetermined frame rate (frames/second).

The imaging control unit 605 also functions as a synchronization control unit that synchronizes the output timing of image data from the imaging device 603 in cooperation with the CPU 611, as will be described later. Note that in this embodiment, the photographing device 6 is not provided with a display, but may be provided with a display section.

Microphone 608 converts sound into sound (signal) data. The sound processing unit 609 takes in audio data output from the microphones 608a, 608b, and 608c through the I/F bus, mixes these audio data, and performs predetermined processing. The sound processing unit 609 also determines the direction of the sound source (speaker) from the sound level (volume) input from the microphones 608a, 608b, and 608c.

The speaker 619 converts input audio data into audio. Microphone 608 and speaker 619 are examples of input/output means. The input/output means do not need to be integrated into one configuration, and the input means and output means may be configured separately.

The CPU 611 controls the overall operation of the photographing device 6 and executes necessary processing. The ROM 612 stores various programs for operating the photographing device 6. The SRAM 613 and DRAM 614 are work memories that store programs executed by the CPU 611, data being processed, and the like. In particular, the DRAM 614 stores image data that is currently being processed by the image processing unit 604 and data of processed equirectangular projection images.

The operation unit 615 is a general term for operation buttons such as the imaging start button 615a. By operating the operation unit 615, the user not only starts imaging and recording, but also turns the power ON/OFF, establishes a communication connection, and inputs settings such as various imaging modes and imaging conditions.

The external device connection I/F 616 is an interface for connecting various external devices. External devices in this case include, for example, the display device 2, a PC (Personal Computer), a display, a projector, and the like. The external device connection I/F 616 may include, for example, a USB terminal, an HDMI (registered trademark) terminal, or the like. The video data and image data stored in the DRAM 614 are transmitted to an external terminal via this external device connection I/F 616 or recorded on external media. In addition, using a plurality of external device connection I/Fs 616, for example, while transmitting and recording image information captured and acquired by the photographing device 6 to the PC via USB, the PC can transmit video (for example, a remote conference) from the PC to the photographing device 6. Image information displayed by an application) may be acquired and further transmitted from the photographing device 6 via HDMI to other external devices (display device 2, display, projector, etc.) for display.

The communication unit 617 communicates with the cloud server via the Internet using wireless communication technology such as Wi-Fi via the antenna 617a provided in the photographing device 6, and transmits the stored video data and image data to the cloud server. You can. Further, the communication unit 617 may be able to communicate with nearby devices using short-range wireless communication technology such as BLE (Bluetooth Low Energy.registered trademark) and NFC.

The audio sensor 618 is a sensor that acquires 360° audio information in order to identify from which direction a loud audio is input in 360° around the photographing device 6 (horizontal plane). The sound processing unit 609 identifies the strongest direction based on the input 360° audio parameters and outputs the 360° audio input direction.

Note that other sensors (azimuth/acceleration sensor, GPS, etc.) may calculate the azimuth, position, angle, acceleration, etc., and may be used for image correction and position information addition.

The image processing unit 604 also creates a wide-field image using the following method. The CPU 611 performs predetermined camera image processing such as Bayer conversion (RGB complementation processing) on RAW data input from an image sensor that inputs spherical images to create a fisheye image (a curved image). Furthermore, DeWarp processing (distortion correction processing) is performed on the created fisheye image (curved surface image) to create a wide-field image (flat image) that captures 360° of the surroundings of the photographing device 6.

The CPU 611 creates a speaker image using the following method. The CPU 611 creates a speaker image by cutting out the speaker from a wide-field image (flat image) showing 360° of the surrounding area. The CPU 611 cuts out a speaker image from the wide-field image, using the audio input direction specified from the 360° output using the audio sensor 618 and the sound processing unit 609 as the speaker's direction. At this time, the method of cutting out a human image from the audio input direction is to cut out a 30° area centered on the specified audio direction from 360°, perform face detection within that area, and then cut out the image. Among the cut out speaker images, the CPU 611 further identifies speaker images for a specific number of people (such as 3 speakers) who have recently spoken.

[Functional configuration of communication system]
Next, the functional configurations of the display device 2, the communication terminal 5, and the photographing device 6, which are the main devices (terminals) in the communication system 1, will be explained using FIG. FIG. 8 is a functional configuration diagram of a display device, a communication terminal, and a photographing device.

<Functional configuration of display device>
As shown in FIG. 8, the display device 2 includes a transmitting/receiving section 21, a receiving section 22, and a display control section 24. Each of these units is a function realized by instructions from the CPU 201 or (and) the GPU 212 according to a program stored in the RAM 203 or the like. Furthermore, the display device 2 includes a storage unit 29 constructed from a RAM 203 and/or an SSD 204.

Among these, the transmitting/receiving unit 21 transmits and receives various data (information) to and from the communication terminal 5. Note that the transmitting/receiving unit 21 receives a matrix type two-dimensional code (hereinafter referred to as a “two-dimensional code”) from the communication terminal 5, and therefore serves as a code information receiving unit.

The reception unit 22 receives various selections or inputs from participants (users).

The display control unit 24 causes a display 280 serving as a display unit to display various images or information. The display control unit 24 also includes a web browser function.

A two-dimensional code is stored in the storage unit 29. This two-dimensional code is identification information for identifying the display device 2, and is an example of code information. The code information also includes a barcode. Furthermore, the code information also includes specific figures, specific icons, specific characters, and the like. The identification information also includes predetermined content information (for example, the name of the destination display device) indicating the predetermined content. The two-dimensional code also includes instruction information to detect a rectangular area.

Note that the storage unit 29 does not necessarily need to store the two-dimensional code. For example, if the manufacturer or seller of the display device 2 and the photographing device 6 are the same, it is possible to store the two-dimensional code in the storage unit 29 in advance, but if they are different, the two-dimensional code There are cases where the two-dimensional code is not stored in the section 29.

<Functional configuration of communication terminal>
As shown in FIG. 8, the communication terminal 5 includes a transmitting/receiving section 51, a receiving section 52, and a display control section 54. Each of these units is a function realized by instructions from the CPU 501 according to a program stored in the RAM 503 or the like.

Among these, the transmitting/receiving section 51 transmits and receives various data (information) to and from the display device 2 or the photographing device 6. Note that the transmitting/receiving section 51 transmits the two-dimensional code to the display device 2, so it plays a role as a code information transmitting section. Further, the transmitting/receiving unit 51 transmits and receives data (information) to and from the communication terminal 10, which is the other party of video distribution, via the communication network 100 and the communication control server 7.

The reception unit 52 receives various selections or inputs from participants (users).

The display control unit 54 causes the display 506 to display various images or information. The display control unit 54 also includes a web browser function.

The storage unit 59 may store a two-dimensional code. This two-dimensional code is identification information for identifying the display device 2 of the transmission destination. The two-dimensional code stored in the storage unit 59 is generated by a dedicated application installed in the communication terminal 5. This two-dimensional code is used when no two-dimensional code is stored in the storage unit 59 of the display device 2, so the information included in both two-dimensional codes (the above-mentioned predetermined content information and instruction information) is basically is the same as Note that if the two-dimensional code is not stored in the storage section 29 of the display device 2, the transmitting/receiving section 51 of the communication terminal 5 transmits the two-dimensional code stored in the storing section 59 to the transmitting/receiving section 21. can be sent.

Furthermore, it is also possible to connect a projector to the communication terminal 5 and project and display the two-dimensional code on a screen or the like, without using the display device 2. In this case, video distribution of a meeting or the like is performed using materials displayed on a screen or the like.

<Functional configuration of imaging device>
As shown in FIG. 8, the photographing device 6 includes a transmitting section 61, a receiving section 62, an imaging section 63, a wide-field image generating section 64, a code information detecting section 65, an instruction information detecting section 66, a rectangular area detecting section 67, a composition It has an image generation section 68 and a predetermined content information detection section 69. Each of these units is a function realized by instructions from the CPU 611 according to a program stored in the SRAM 613 or the like.

Among these, the transmitter 61 transmits various data (information) to the display device 2 . For example, the transmitter 61 transmits a predetermined composite image (or specific composite image) generated by the composite image generator 68 to the external communication terminal 5. The predetermined composite image and specific composite image will be described later.

The reception unit 62 receives various selections or inputs from the user.

The imaging unit 63 images a subject, scenery, etc., and obtains fisheye image data.

The wide-field image generation unit 64 generates a wide-field image from the fisheye image captured by the imaging unit 63.

The code information detection unit 65 detects a two-dimensional code included in the wide-field image.

When the code information detection unit 65 detects a two-dimensional code, the instruction information detection unit 66 detects instruction information to the effect that a rectangular area is to be detected from the two-dimensional code.

When the instruction information detection section 66 detects instruction information, the rectangular area detection section 67 detects a predetermined rectangular area outside the two-dimensional code 8 in the wide-field image. Since the frame of the display 280 is rectangular, the rectangular area detection unit 67 essentially detects the display 280. Note that "outside the two-dimensional code 8" indicates an area outside the two-dimensional code 8 including the two-dimensional code 8, as shown in FIG.

When a predetermined rectangular area is detected by the rectangular area detection unit 67, the composite image generation unit 68 synthesizes at least a predetermined rectangular area image indicating the predetermined rectangular area in the wide-field image and the wide-field image. A predetermined composite image is generated. Furthermore, the composite image generation unit 68 fits the predetermined rectangular area image into a predetermined fixed display area 440 (see FIG. 13) within the predetermined composite image. Note that even if the two-dimensional code is no longer detected by the code information detection unit 65 after the composite image generation unit 68 generates the predetermined composite image, the composite image generation unit 68 will generate the predetermined rectangular area image and the wide-field image. Continue to generate a predetermined composite image by combining the images.

On the other hand, when the instruction information detection section 66 detects instruction information, but the rectangular area detection section 67 does not detect a predetermined rectangular area, the composite image generation section 68 detects at least two areas within the wide-field image. A specific composite image is generated by combining a specific rectangular area image indicating a specific rectangular area calculated based on the position and size of the dimensional code and a wide field of view image. In this case as well, the composite image generation unit 68 fits the specific rectangular area image to a predetermined fixed display area 440 (see FIG. 13) within the specific composite image. Furthermore, even if the two-dimensional code is no longer detected by the code information detection unit 65 after the composite image generation unit 68 generates the specific composite image, the composite image generation unit 68 can detect the specific rectangular area image and the wide-field image. Continue to generate a specific composite image by combining the images.

When the code information detection unit 65 detects a two-dimensional code, the predetermined content information detection unit 69 detects predetermined content information indicating predetermined content from the two-dimensional code. The predetermined content information includes display device identification information for identifying the display device 2 used as a set with the photographing device 6. The display device identification information is, for example, information indicating "electronic blackboard No. 1" displayed in the display area 453 of FIG. 13. When the predetermined content information detecting section 69 detects the predetermined content information, the transmitting section 61 transmits the predetermined content information to the external communication terminal 5.

[Communication system processing]
Next, the processing of the communication system 1 including the image processing method by the photographing device 6 will be explained using FIGS. 9 to 13. FIGS. 9 and 10 are flowcharts showing the process of specifying a fixed display area by the photographing device. FIGS. 11 and 12 are diagrams showing two-dimensional codes displayed on the display of the display device.

First, as shown in FIG. 2, as a premise for video distribution between base α and base β, as shown in FIG. 11, display device 2 displays two-dimensional code 8 on display 280 at base α.

There are two ways to display the two-dimensional code 8. As a first display method, the display control unit 24 of the display device 2 reads the two-dimensional code data stored in the storage unit 29 and displays the two-dimensional code 8 on the display 280. As a second display method, when the display device 2 does not store the two-dimensional code on its own, the transmitting/receiving section 51 of the communication terminal 5 reads the data of the two-dimensional code from the storage section 59 and displays it on the display device 2. The display controller 24 of the display device 2 displays the two-dimensional code 8 on the display 280 based on the two-dimensional code data received by the transmitter/receiver 21 . Note that if the two-dimensional code data is not stored in either the display device 2 or the communication terminal 5, the display device 2 cannot display the two-dimensional code 8.

S11: In FIG. 9, the wide-field image generation unit 64 first generates a wide-field image from the fisheye image captured by the imaging unit 63.

S12: The code information detection unit 65 detects a two-dimensional code included in the wide-field image. If there is a two-dimensional code, the process advances to S13, and if there is no two-dimensional code, the process in FIG. 9 ends.

S13: When the two-dimensional code is detected by the code information detection unit 65 (S12; YES), the instruction information detection unit 66 detects instruction information to the effect that a rectangular area is to be detected from the two-dimensional code. If the two-dimensional code includes instruction information, the process advances to S14, and if the two-dimensional code does not include instruction information, the process in FIG. 9 ends.

S14: If the instruction information detection unit 66 detects instruction information (S13; YES), the rectangular area detection unit 67 detects a predetermined rectangular area outside the two-dimensional code in the wide-field image. If there is a rectangular area, the process advances to S15, and if there is no rectangular area, the process advances to S16.

S15: If the rectangular area detection unit 67 detects a predetermined rectangular area (S14; YES), the composite image generation unit 68 generates at least a predetermined rectangular area image indicating the predetermined rectangular area in the wide-field image. , and the wide-field image to generate a predetermined composite image. In this case, the composite image generation unit 68 fits the predetermined rectangular area image into the fixed display area 440 (see FIG. 13).

FIG. 13 is a diagram showing a screen displayed by the communication terminal 5 at the base α. As described above, the communication terminal 5 may be one of the PCs of the participants shown in FIG. 2, or may be a dedicated PC. The predetermined composite image generated by the composite image generation unit 68 is displayed on the screen 400 displayed by the communication terminal 5 in a display area 410 of an image to be transmitted to the other party of the video distribution at the base β. Thereby, the participants at the base α can understand the content of the image transmitted to the base β side.

Further, the display area 410 includes a display area 420 for wide-field images, display areas 431 to 433 of people in the wide-field images displayed in the order of each utterance, and a fixed display area 440 for a predetermined rectangular area image. In display areas 431 to 433, the latest speaker is displayed in display area 431, the previous speaker is displayed in display area 432, and the two previous speaker is displayed in display area 433. Then, each time a new speaker appears, the displayed person is moved to the lower display area and displayed. Other icons 451 and display areas 452 and 453 on screen 400 will be described later.

Note that even if the two-dimensional code is no longer detected by the code information detection unit 65 after the composite image generation unit 68 generates the predetermined composite image, the composite image generation unit 68 will generate the predetermined rectangular area image and the wide-field image. Continue to generate a predetermined composite image by combining the images. Further, after the code information detection unit 65 no longer detects the two-dimensional code, if the code information detection unit 65 detects a two-dimensional code again, the processing from S12 onwards is started again. Thereby, readjustment of the predetermined rectangular area image is possible.

S16: If the predetermined rectangular area is not detected by the rectangular area detection unit 67 (S14; NO), the composite image generation unit 68 at least performs a detection based on the position and size of the two-dimensional code within the wide-field image. A specific combined image is generated by combining the specific rectangular area image indicating the calculated specific rectangular area and the wide-field image. For example, as shown in FIG. 12, when a participant displays a two-dimensional code at a predetermined position on the screen 11 from a projector connected to the communication terminal 5 without using the display device 2, the rectangular area detection unit 67, the frame (rectangular area) of the display 280 of the display device 2 may not be detected. In such a case, the composite image generation unit 68 generates a specific rectangular area image to be displayed in the fixed display area 440 of FIG. 13 based on the position of the displayed two-dimensional code and the predetermined size of the two-dimensional code. A specific rectangular area 12 is calculated. For example, the composite image generation unit 68 calculates a specific rectangular area 12 having a width that is six times the width of the two-dimensional code 8 and a height that is three and a half times the height of the two-dimensional code 8. Thereby, the composite image generation unit 68 can estimate the specific rectangular area 12 related to the specific rectangular area image to be displayed in the fixed display area 440, so that it is possible to fit the specific rectangular area image to the fixed display area 440. can. Note that the horizontal to vertical aspect ratio may be 4:3, 16:9, etc. in addition to 6:3.5.

Note that even if the two-dimensional code is no longer detected by the code information detection unit 65 after the composite image generation unit 68 generates the specific composite image, the composite image generation unit 68 will generate the specific rectangular area image and the wide-field image. Continue to generate a specific composite image by combining the images. Further, after the code information detection unit 65 no longer detects the two-dimensional code, if the code information detection unit 65 detects a two-dimensional code again, the processing from S12 onwards is started again. Thereby, readjustment of the specific rectangular area image is possible.

Further, the photographing device 6 performs the process shown in FIG. 10 in parallel with the process shown in FIG.

S21: In the process of S12 in FIG. 9, if the code information detection unit 65 detects a two-dimensional code (S12; YES), predetermined content information indicating predetermined content is detected from the two-dimensional code. As described above, the display device identification information is, for example, information indicating "electronic blackboard No. 1" displayed in the display area 453 of FIG. 13. If the two-dimensional code includes the predetermined content information, the process advances to S22, and if the two-dimensional code does not include the predetermined content information, the process in FIG. 10 ends.

S22: When the predetermined content information is detected by the predetermined content information detection unit 69 (S21; YES), the transmitting unit 61 transmits the predetermined content information to the external communication terminal 5. As shown in FIG. 13, the screen 400 displayed by the communication terminal 5 includes, in addition to the display area 410, an icon 451 for reading conference information from the scheduler, which indicates the start date and time, end date and time, and location of the conference. It includes a display area 452 for displaying conference information, and the above-mentioned display area 453. As a result, participants at the base α can grasp most of the information regarding video distribution of a conference, etc. by looking at the screen 400 in FIG. 13. In addition, the screen 400 shown in FIG. 13 is recorded by the communication terminal 5 and also serves as minutes of the meeting.

This concludes the explanation of the processing of the communication system 1 including the image processing method by the photographing device 6.

[Other embodiments]
A modification example in which the display device 2 displays the two-dimensional code 8 will be described below.

FIG. 14 is an example of the two-dimensional code 8 displayed by the display device 2. In FIG. 14, one large two-dimensional code 8 is displayed. When the two-dimensional code 8 is large, the photographing device 6 can easily detect the two-dimensional code 8. Below, a method of displaying the two-dimensional code 8 will be explained.

FIG. 15 is an example of the position of the two-dimensional code 8 displayed by the display device 2. As described above, the two-dimensional code 8 includes identification information and the like. On the desk 110 on which the photographing device 6 is placed, there is an obstacle 3 between the photographing device 6 and the display device 2. Note that in detecting the two-dimensional code 8, the photographing device 6 does not need to be a spherical camera. The display device 2 displays the identification information above the center line 320 of the display screen of the display device 2 in the vertical direction. By doing so, even if there is an obstacle 3 between the photographing device 6 and the display device 2, the photographing device 6 can easily image the identification information.

Note that, as shown in FIG. 16, the display device 2 may display the center of the two-dimensional code 8 above the center line 320 of the display screen of the display device 2 in the vertical direction.

Further, as shown in FIG. 17, the display device 2 may move the two-dimensional code 8 according to time. In FIG. 17, the two-dimensional code 8 is moving from left to right. The display device 2 may move while displaying the two-dimensional code 8, or it may display -> erase -> change the location and re-display. Thereby, even if the position of the obstacle 4 is uncertain, the imaging device 6 can easily capture the device identification information. Furthermore, even when the display device 2 displays the two-dimensional code 8 in a small size to reduce the feeling of pressure on the user, it becomes easier for the imaging device 6 to capture the image.

Furthermore, the display device 2 may change the size of the two-dimensional code 8 while moving the two-dimensional code 8.

Further, as shown in FIG. 18, the display device 2 may display a plurality of two-dimensional codes 8 at the same time. Thereby, even if a part of the two-dimensional code 8 is hidden by the obstacles 3 and 4, the imaging device 6 can easily image the remaining two-dimensional code 8. Note that the plurality of two-dimensional codes 8 may all store the same information, or may store mutually different information.

Further, as shown in FIG. 19, the display device 2 may display the two-dimensional code 8 next to (in close proximity to) the menu 71. The menu 71 is arranged across the right end in the vertical direction, and is displayed above the center line 320 of the display screen of the display device 2 in the vertical direction, similar to FIG. 15 .

By displaying the two-dimensional code 8 near the menu 71, the user is less likely to feel uncomfortable. Users can use more screen space.

Further, as shown in FIG. 20, the display device 2 may display the two-dimensional code 8 within the menu 71. Compared to FIG. 19, it is easier to eliminate the feeling of pressure, and the user can use the screen more widely.

FIG. 21 is a diagram illustrating a case where the camera included in the photographing device 6 is a hemispherical camera. By capturing images with a hemispherical camera with a wide horizontal field of view, the two-dimensional code 8 can be found more easily.

Note that a barcode instead of the two-dimensional code 8 can be displayed in the same manner as in FIGS. 14 to 21. FIG. 22 is an example of the barcode 9.

The barcode 9 has lower robustness against tilt and the like than the two-dimensional code 8. Therefore, the code information detection unit 65 cuts out the black and white pattern and corrects the skew angle and pitch angle. The code information detection unit 65 emphasizes the edges of the black bar. The code information detection unit 65 detects the barcode 9 displayed by the display device 2 by pattern matching the cut out image with a pattern from the start character to the rightmost stop character registered as the pattern of the barcode 9. do.

Furthermore, in FIGS. 14 to 22, an example in which a two-dimensional code 8 or a barcode 9 is displayed has been described, but the display device 2 displays identification information such as alphabets and numbers, and the code information detection unit 65 performs OCR ( The identification information may be recognized by Optical Character Recognition/Reader) processing.

[Main effects of the embodiment]
As described above, according to the present embodiment, the photographing device 6 can be automatically detected by taking advantage of the fact that the display 280 of the display device 2, which is mainly treated as an area of interest, is rectangular. Then, the area of interest within the wide-field image is enlarged and set as the fixed display area 440. This has the effect that the work of the user of the video distribution source can be reduced.

[Other embodiments]
In the embodiment described above, when the instruction information detection unit 66 detects instruction information to detect a rectangular area, the rectangular area detection unit 67 detects a predetermined rectangular area outside the code information in the wide-field image. , but is not limited to this. For example, the code information does not include "instruction information to detect a rectangular area" and the code information indicates specific predetermined information (manufacturing or sales company name, product name, etc.). , when the code information detection unit 65 detects code information, the rectangular area detection unit 67 may detect a rectangular area around the code information in the wide-field image.

Further, the rectangular area detection unit 67 may not only detect a rectangular area but also generate a rectangular area image based on the detected rectangular area. In this case, the rectangular area detection section 67 also serves as a rectangular area generation section.

Note that other embodiments can be summarized as follows.

<Additional notes>
A photographing device that outputs an image,
a wide-field image generation unit that generates a wide-field image from a fisheye image obtained by imaging;
a code information detection unit that detects code information included in the wide-field image;
a rectangular area generation unit that detects a rectangular area outside the code information in the wide-field image and generates a rectangular area image when the code information is detected by the code information detection unit;
a composite image generation unit that generates a predetermined composite image by combining the rectangular area image and the wide-field image;
a transmitter that transmits the predetermined composite image generated by the composite image generator to the outside;
A photographing device characterized by having the following.

[Other application examples]
Although the embodiments of the present invention have been described above, the present invention is not limited to these embodiments in any way, and various modifications and substitutions can be made without departing from the gist of the present invention.

Each function of the embodiments described above can be realized by one or more processing circuits. Here, the term "processing circuit" as used herein refers to a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, or a processor designed to execute each function explained above. This includes devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules.

Furthermore, each of the units 61 to 63 and 65 to 68 of the photographing device 6 can be realized by a computer by executing a program. Further, this program can also be provided by being recorded on a (non-temporary) recording medium.

1 Communication system 2 Display device 5 Communication terminal 6 Photographing device 21 Transmitting/receiving section (an example of a code information receiving section)
22 Reception section 24 Display control section 29 Storage section 51 Transmission/reception section (an example of a code information transmission section)
52 Reception section 54 Display control section 59 Storage section 61 Transmission section 62 Reception section 63 Imaging section 64 Wide-field image generation section 65 Code information detection section 66 Instruction information detection section 67 Rectangular area detection section 68 Composite image generation section 69 Predetermined content information detection Department

Meeting Owl Pro User Manual <https://www.sourcenext.com/produce/app/manual/meetingowl/JP/index.html#!/06_02>

Claims (12)

A photographing device that outputs images for video distribution,
a wide-field image generation unit that generates a wide-field image from a fisheye image obtained by imaging;
a code information detection unit that detects code information included in the wide-field image;
an instruction information detection section that detects instruction information to the effect that a rectangular area is to be detected from the code information when the code information is detected by the code information detection section;
a rectangular area detection unit that detects a predetermined rectangular area outside the code information in the wide-field image when the instruction information is detected by the instruction information detection unit;
When the predetermined rectangular area is detected by the rectangular area detection unit, a predetermined composite image is generated by combining a predetermined rectangular area image indicating the predetermined rectangular area in the wide-field image and the wide-field image. a composite image generation unit that
a transmitter that transmits the predetermined composite image generated by the composite image generator to the outside;
A photographing device characterized by having the following. The photographing device according to claim 1, wherein the composite image generation unit fits the predetermined rectangular area image into a predetermined fixed display area within the predetermined composite image. Even if the code information is no longer detected by the code information detection unit after the composite image generation unit generates the predetermined composite image, the composite image generation unit may The photographing device according to claim 1, further comprising the step of continuing to generate the predetermined composite image by combining the image with a visual field image. The imaging device according to any one of claims 1 to 3,
a predetermined content information detection unit that detects predetermined content information indicating predetermined content from the code information when the code information is detected by the code information detection unit;
The photographing device is characterized in that when the predetermined content information is detected by the predetermined content information detection unit, the transmitting unit transmits the predetermined content information to the outside. 5. The photographing device according to claim 4, wherein the predetermined content information includes display device identification information for identifying a display device used as a set with the photographing device. When the instruction information is detected by the instruction information detection section, but the predetermined rectangular area is not detected by the rectangular area detection section, the composite image generation section detects the code in the wide-field image. The imaging device according to claim 1, wherein a specific composite image is generated by combining a specific rectangular area image indicating a specific rectangular area calculated based on the position and size of the information and the wide-field image. The photographing device according to claim 6, wherein the composite image generation unit fits the specific rectangular area image into a predetermined fixed display area within the specific composite image. Even if the code information is no longer detected by the code information detection unit after the composite image generation unit generates the specific composite image, the composite image generation unit may The photographing device according to claim 6, further comprising the step of continuing to generate the specific composite image by combining it with a visual field image. The imaging device according to claim 1 or 6;
a rectangular display device;
A communication system having:
The display device includes:
a storage unit that stores the code information;
a display control unit that displays the code information on a display unit;
A communication system comprising: The imaging device according to claim 1 or 6;
A communication terminal,
a rectangular display device;
A communication system having
The communication terminal is
a storage unit that stores the code information;
a code information transmitter that transmits the code information to the display device;
has
The display device includes:
a code information receiving unit that receives the code information;
a display control unit that displays the code information on a display unit;
A communication system comprising: An image processing method executed by a photographing device that outputs images for video distribution, the method comprising:
wide-field image generation processing that generates a wide-field image from a fisheye image obtained by imaging;
code information detection processing for detecting code information included in the wide-field image;
instruction information detection processing for detecting instruction information to detect a rectangular area from the code information;
rectangular area detection processing that detects a predetermined rectangular area outside the code information in the wide-field image based on the instruction information detection processing;
a composite image generation process that generates a predetermined composite image by combining a predetermined rectangular area image indicating the predetermined rectangular area in the wide-field image and the wide-field image;
a transmission process of transmitting the predetermined composite image generated by the composite image generation process to the outside;
An image processing method characterized by performing the following. A program that causes a computer to execute the method according to claim 11.

Images