TWI739585B - Full fov conference camera device - Google Patents

Abstract

The purpose of the present invention is to improve the conventional network video conference system not only lacking the function of merging multiple images as a concise integrating image, but the components of system are dispersion without one piece integration, resulting in inconvenience working experience. Therefore, the present invention proposes a novel Full FOV Conference Camera Device, the device structure comprises a plurality of camera unit and an audio/video processing module, used for acquiring and editing the conference video as an integrating image. The audio/video processing module further comprises a presentation mode, which edit the image of conference participants with the required background image into a composite image. to increase the efficiency of telecommute.Wherein a plurality of Full FOV Conference Camera Device deployed on different location could link with each other by cloud network for composing a cloud conference system to increase the efficiency of telecommute.

Description

Full-view conference photography device

The present invention relates to a full-view conference photographing device. More specifically, it is a camera with multiple photographic lenses, which can aim at a 360° viewing angle of a meeting room, with target tracking (ROI) and image processing, so that conference speakers can present presentations In the Presentation Mode, improve the existing online conference photography system.

In recent years, due to the rise of remote offices, in order to enable workers dispersed at remote locations to quickly participate in work meetings even if they are not working in a fixed location, online video conferencing has emerged as the times require. The so-called online video conference is a kind of terminal that is separated from many places, through video, voice and data input and output equipment and network connection, so that conference participants can reach the same meeting place, quickly and reliably exchange data and communicate opinions. A system. In order to achieve the purpose of multi-party communication, the meeting rooms are equipped with cameras, screens, monitors, microphones, and speakers, and are connected to each other through terminal devices. Generally speaking, in addition to taking pictures of the participants and the host, each terminal must also be able to transmit written and electronic data for the purpose of communication.

In the past conference video system, if the terminal in one place or multiple places contains multiple conference participants, on the screen of each video conference, there will be image windows taken by different cameras to show the participants of each conference. However, this screen display method does not take into account the interactive relationship and environmental connection between the participants, resulting in a lack of a sense of connection between the participants, which often results in participants not participating in the same meeting, or even during the meeting. The illusion that it's nothing to do. In addition, the past conference video system usually lacks integration. If there are more people attending the meeting at the same meeting place, multiple cameras, microphones, and transmission lines are scattered all over the place. In addition to the aforementioned screen display problems, whenever the meeting is When the speaker leaves the camera's field of view, the meeting may also be interrupted, requiring manual adjustment of the camera's position, which makes the process of carrying, setting up, and using the previous video system seem cumbersome and inconvenient.

Based on the above problems, in the current market, some manufacturers have launched an integrated conference video system, which integrates the above-mentioned camera, microphone, transmission line, transmission module, etc. into a portable device to improve its installation. And ease of use. For example, in Japanese patent JP2018521593A proposed by Owl Labs, it shows an integrated video device with several camera lenses and microphones, which can automatically track during the meeting according to the audio data obtained by the camera lens and microphone. The function of participants or moderators, and the function of dividing or stitching images according to the number of participants in the meeting. For another example, the Japanese patent JP2011244455A proposed by Polycom Corporation also shows a conference video system, and has the functions of integrating or splitting images for conference participants and tracking the host.

However, due to the different environment of the meeting place, which may include small meeting rooms or larger lecture halls, there is still no meeting video system on the market that has the ability to target the resolution and frame of the image depending on the meeting place. Select the frame rate (Frames per second, fps), and emphasize that when the photographic screen is tracking the host or participant (ROI, Region of Interest), it can have multiple stitching and collage images according to the different topics. Mode function. In addition, the existing ROI function generally uses a camera lens that is moved by a mechanical motor. The camera needs to be physically rotated during the tracking process, which may cause the image to be unstable. What's more, many conference video systems still adopt a decentralized design. It has not been integrated into a one-piece form, so based on the above shortcomings, the existing video system still has room for further improvement.

In order to solve the above problems, the present invention proposes a full-view conference camera device. A processing module is used to coordinate the operation and management of the system. The detailed structure includes a housing for carrying or disposing related electronic components and mechanisms. ; N photographing units to shoot conference images; the audio-visual processing module, according to the needs of the application, cuts or collages the conference images taken by the aforementioned n photographing units. Among them, the audio-visual processing module further includes: a target tracking unit to track the location of the meeting participants; and a segmentation unit to segment the meeting image into Panorama mode and Focus mode according to the needs of the application , And multi-screen mode. Among them, the multi-screen mode can divide the image into a top-down mode (Top-Down mode) with a top-down split screen, and a grid mode (Grid Mode) with a multi-person screen. Among them, according to the optimized embodiment of the present invention, the number of photographing units is 4 (that is, n=4). In addition, in this case, for ease of description, the first photographing unit and the second photographing unit , The third photography unit, They can be collectively referred to as photographic units, which are described here first. The above n photographing units are arranged equidistantly on a virtual circle on the housing.

According to the content of the present invention, the audio-visual processing module includes a presentation unit, wherein a presentation mode (Presentation Mode) is installed in the presentation unit. Whiteboards, projection screens, or flat walls in the meeting room) are spliced into presentation images.

According to the content of the present invention, when an online conference starts, at least two full-view conferences can be configured in different geographic areas (for example: the first lecture hall, the second lecture hall, the third lecture hall, or the homes of conference participants, etc.) The camera device can form a cloud conference system, and connect each terminal with the cloud network through the transmission module. Through the transmission module, the cloud conference system integrates the camera unit and audio-visual processing module of each full-view conference camera device. So that each terminal can receive the conference video, and the corresponding electronic data and written data.

According to the content of the present invention, in the Panorama mode, a conference image with an angle range of 90°-360° captured by n photographing units by a dividing unit is processed into a panoramic image that can be displayed on a plane. Wherein, in the embodiment of the present invention, the field of view (FOV) of each photographing unit is 30°-100°.

According to the content of the present invention, the focus mode is to track and focus on the location of the participants of each meeting by the target tracking unit, and then process the focused images within the predetermined range of each participant by the segmentation unit After rendering.

According to the content of the present invention, the appearance of the full-view conference camera device is integrally formed, wherein the n camera units adopt an integrated structure to improve the lack of scattered cameras in the conventional conference video system.

According to an embodiment of the present invention, the camera unit is set to a fixed type, with optical zoom or digital zoom for up, down, left, and right adjustment and zooming of conference images (left and right adjustment Pan; up and down adjustment Tile; zoom zoom. English abbreviation: PTZ mode). To improve the conventional conference video system, when the camera needs to use PTZ to adjust the conference image, a mechanical motor is required, which may cause the camera to shake and make the conference image unstable.

According to the content of the present invention, the full-view conference camera device includes a transmission module to transmit conference images, panoramic images, presentation images, and focused images to external terminals. Wherein, in an embodiment of the present invention, the specification of the transmission module can be but not limited to USB Mini, USB Micro, USB Type A, USB Type B, USB Type C, LPT, RS232, PS/2, or any of the above Combination, and can replace any required transmission specifications according to the needs of the application.

The above descriptions are used to illustrate the purpose, technical means and achievable effects of the present invention. Those familiar with this technology in the relevant field can get a clearer understanding of the present invention through the demonstration of the following examples and accompanying schematic descriptions and the scope of patent applications. invention.

100: Full-view conference photography installation

200: Cloud Conference System

201: Processing Module

203: The first photography unit

205: The second photography unit

207: The third photography unit

209: Audio and Video Processing Module

209a: Target tracking unit

209c: segmentation unit

209e: Presentation mode unit

211: Transmission Module

213: Shell

300a: Panorama mode

300b: Multi-picture mode

300c: Split mode

300d: Frame mode

300f: Focus mode

500a: Presentation mode

The detailed description of the present invention and the schematic diagrams of the embodiments described below should make the present invention more fully understood; however, it should be understood that this is only used as a reference for understanding the application of the present invention, and does not limit the present invention to a specific implementation. In the case.

Figure 1 shows the front appearance of a full-view conference camera device.

Figure 2 illustrates the system architecture of a full-view conference camera device.

Figure 3 shows the detailed system architecture of the segmentation unit.

FIG. 4A illustrates that the dividing unit processes the conference image into a panoramic image in a panoramic mode.

FIG. 4B illustrates that the dividing unit processes the conference video into a divided video with top and bottom divided screens in a divided mode.

FIG. 4C illustrates that the dividing unit processes the conference video into a divided video in the form of a frame in a dividing mode.

Figure 4D illustrates that the segmentation unit processes the conference image into a focused image in a focused mode.

Figure 5 shows the detailed system architecture of the presentation mode unit.

Fig. 6A illustrates the image processing method of the presentation unit for the presentation mode.

FIG. 6B illustrates the image processing method of the presentation unit for the presentation mode.

Fig. 6C illustrates the image processing method of the presentation unit for the presentation mode.

Figure 6D illustrates the use of the cloud conference system for online conferences in various locations through the cloud network.

The present invention will be described in detail with preferred embodiments and viewpoints. The following description provides specific implementation details of the present invention, so that the reader can thoroughly understand the implementation of these embodiments. However, those skilled in the field must understand that the present invention can also be implemented without these details. In addition, the present invention can also be applied and implemented by other specific embodiments. The details described in this specification can also be applied based on different needs, and various modifications or changes can be made without departing from the spirit of the present invention. . The present invention will be described in terms of preferred embodiments and viewpoints. Such description is to explain the structure of the present invention, and is only for illustration and not to limit the scope of patent application of the present invention. The terms used in the following description will be interpreted in the broadest reasonable manner, even if they are used in conjunction with the detailed description of a specific embodiment of the present invention.

The first purpose of the present invention is to improve the conventional video conference system that can only display the images taken by the cameras deployed in various places in the conference room, or by the cameras of each terminal, instead of displaying each participant of the taken meeting. The function of integrating the meeting video with the host makes the participants lack the sense of connection with each other, and does not have the illusion of participating in the same meeting. The second purpose of the present invention is to improve the It is known that the conference video system lacks integration. If there are more people participating in the meeting at the same meeting place, multiple cameras, microphones, and transmission lines will be scattered everywhere. In addition to the aforementioned screen display problems, whenever the speaker of the meeting leaves the camera In the range of angle of view, there are few functions related to ROI (Region of Interest), and the position of the camera needs to be adjusted manually, which makes the process of carrying and erecting the system inconvenient. The third object of the present invention is to improve the ROI function of the camera attached to the existing conference video system. The shooting angle of the camera is usually adjusted by a mechanical motor. During the tracking process, the lens needs to be physically rotated, which may lead to instability of the image. Finally, the fourth objective of the present invention is to improve the image resolution and frame rate of conventional cameras, so that during the online meeting, the quality of the meeting images can be adjusted according to different usage conditions, so as to achieve communication or exchange of opinions through online meetings. Data, increase the efficiency of remote office. To achieve the above objectives, the detailed technical means and implementation manners of the present invention will be described in detail later.

Please refer to FIG. 1. In order to solve the above-mentioned problems, the present invention proposes a full-view conference camera device (100) with an integrated structure. A processing module (201) is used to couple the various components in the system to coordinate the operation of the system. And management. Please refer to FIG. 2, the detailed system architecture of the present invention includes: a housing (213) for carrying or configuring the electronic components and mechanisms of the system; n photographing units for shooting conference images; and an audio-visual processing module (209) ), according to the needs of the application, crop or collage the conference images taken by the aforementioned n photographing units. Wherein, the n photographing units are arranged equidistantly on the virtual horizontal circumference of the housing (213), so that they have the same height and are arranged equidistantly. The section of the housing (213) can be square, rectangular, circular, Polygons and so on. Wherein, the audio-visual processing module (209) further includes: a target tracking unit (209a) to track the location of the conference participants; and a segmentation unit (209c) to divide the conference image into a panoramic mode (300a) according to the needs of the application , Panorama mode), focus mode (300c, Focus mode), and multi-screen mode (300b). Among them, the multi-screen mode can divide the image into a top-down mode (Top-Down mode) with top and bottom split screens, and a grid mode (Grid Mode) with multi-person screens. In addition, in the present invention, the processing module (201) usually includes a processing chip (Microprocessor Control Unit, MCU), memory, display card, network card, operating system and application programs, etc., which are generally known It can be connected to each other in ways to perform calculation, temporary storage, display and data transmission, and provide functions such as the operation and management coordination of the full-view conference camera device (100). Among them, the type of operating system that can be supported can be, but not limited to, Linux, Windos , Android, Mac OS or iOS, based on the processing module (201) belongs to the generally known architecture, so it will not be repeated here.

Please refer to FIG. 1, according to an embodiment of the present invention, n photographing units are arranged in a ring-shaped arrangement and are evenly arranged on the housing shell (213) of the full-view conference photographing device (100), so that each photographing unit It can include a shooting angle of 360°/n. For example, in a preferred embodiment of the present invention, there are 4 photographing units, and the shooting angle of view of each photographing unit is 90°. Among them, it should be noted that the lens of the camera unit can be selected according to the needs of the application, such as the size of the meeting room or lecture hall that needs to be set up, choose different lens diameters, viewing angles and lens numbers to achieve optimal shooting Effect. For example, in the above-mentioned embodiment, when the full-view conference camera device (100) has 4 camera units, the resolution and corresponding frame rate of the conference video shot by it can reach the following standard: 3840 X 2160, 10-50fps; 3840 X 640, 10-60fps; 1920 X 1080, 10-70fps.

1 and 3, according to the content of the present invention, the audio-visual processing module (209) includes a target tracking unit (209a), providing a full-view conference camera device (100), a target tracking (ROI, Region of Interest, or ROI) Region of interest). In the present invention, the target tracking is to automatically or manually define the faces and bodies of meeting participants or moderators (for example, each image in the photographing unit) in the meeting images taken by n photographing units. The video processing module (209) can continue to use up, down, Adjust and zoom the conference video screen from left and right (Adjust Pan left and right; Adjust Tile up and down; Zoom Zoom. English abbreviation: PTZ mode), so that the split unit (209c) can adjust the display mode of the screen according to the needs of the online meeting to make the target When the tracking unit (209a) performs target tracking and focusing and needs to move the shooting area, there is no need to rotate the lens through a mechanical motor, so that it can achieve better picture stability and achieve the purpose of improving the image quality of the meeting.

In view of the above, please refer to FIG. 3. In a preferred embodiment of the present invention, the dividing unit (209c) includes several conference image processing methods to process the conference images through the transmission module (211) as required by the application. show. The above-mentioned conference video processing methods include: Panorama mode (300a), Focus mode (300f), and multi-screen mode (300b). The multi-screen mode further includes the split mode (Top-Down). mode, 300c), and the grid mode (Grid Mode, 300d). Please refer to Figures 4A-4D, which respectively illustrate the panorama mode (300a), the split mode (300c), and the frame mode (300d), And the processing method of focus mode (300f). Among them, the panoramic mode (300a) is a division unit (209c) that divides the photographing angle range of n photographing units such as the first photographing unit (203), the second photographing unit (205), and the third photographing unit (207). It is a 90°-360° meeting image, that is, the meeting image in the meeting room or lecture hall where the participants and the host are located, and is processed into a panoramic image that can be displayed on a plane. The viewing angle range of each camera unit is It is 30°-100°.

In addition, please refer to FIG. 4B. The segmentation mode (300c) is that after the location of each participant in the meeting is defined by the aforementioned target tracking unit (209a), the segmentation unit (209c) adjusts the panoramic image to an appropriate size Evenly split into two 90°-180° viewing angles, or two or more split images that can be arranged up and down, so that the meeting situation in the meeting room or lecture hall can be seen through one screen, so as to increase the participation of meeting participants. I feel. Next, please refer to FIG. 4C. According to the content of the present invention, the frame mode (300d) is a dividing unit (209c) that further divides the aforementioned panoramic image into 4 or more, with a viewing angle of 50°-90°, side by side and side by side. Split images that overlap top and bottom. Finally, please refer to Figure 4D. The focus mode (300f) described is for when there are a large number of participants in the meeting, the panoramic image can be further enlarged and focused on the participants, and the viewing angle can be reduced, so that the focused image can be maximized. Focusing on the display of participants. According to a preferred embodiment of the present invention, the number of participants that can be displayed in the focus mode (300f) is 6-14, and the maximum number of participants is 14 people. . Among them, the above-mentioned split mode (300c), frame mode (300d), and focus mode (300f) processing methods, in addition to the large number of participants, the screen can display the participants as much as possible, but also can not lose the meeting Under the premise of image quality, by cutting out the images that do not need to be displayed or irrelevant, such as clutter, background, and other irrelevant meeting content, it can reduce the network bandwidth occupancy rate and reduce the storage of meeting images in the future. space.

Please refer to FIG. 5 and FIGS. 6A-6C. According to the content of the present invention, the audio-visual processing module (209) includes a presentation unit (209e), wherein a presentation mode (500a, Presentation Mode) is installed in the presentation unit (209e). According to an embodiment of the present invention, the function of the presentation mode (500a) is to separate the meeting host, presenter, or participant to be followed from the meeting image in the photographing unit through the target tracking unit (209a). And splicing the conference image of the required background (for example, the whiteboard, projection screen, or flat wall of the conference room) into a presentation image, the presentation mode (500a), and you can choose to place the presenter in the center configuration. For example, when the first photographing unit (203) photographs the host in FIG. 6A and the second photographing unit (205) photographs the background of a whiteboard in a conference room, the presentation unit (209e) can pass through the target tracking unit (209a). ) The host’s image, The image of the whiteboard background of the conference room is separated and merged, so that the final presentation image output to another terminal can include the background of the host and the whiteboard of the conference room, or be combined by the transmission module (211) The inputted written materials, electronic materials, etc. The above-mentioned target tracking unit (209a) will drive the related first photographing unit (203) to move or rotate the lens along with the presenter or host.

In view of the above, in another embodiment of the present invention, please refer to FIG. 6C. When the first photographing unit (203) photographs the host or presenter of the meeting, the second photographing unit (205) and the third photographing unit (207) ) Or the remaining n-1 photography units may be used to shoot other scenes or participants of the meeting, and processed into panoramic images, split scenes, focused images, etc. through the audio-visual processing module (209). At this time, the presentation unit ( 209c) Through the function of the presentation mode (500a), the presentation image can be combined with the aforementioned panoramic image, segmented scene, and focused image to be output as a single screen. Take the description in Figure 6C as an example. The meeting host, the whiteboard in the meeting room, the electronic data of the meeting, and the surround view image on the whiteboard are all in the presentation mode (500a), which combines n photographic units of the meeting The image and the electronic data in the transmission module (211) are combined into one screen, and the final presentation image is obtained.

According to the content of the present invention, the specifications of the transmission module (211) can be but not limited to USB Mini, USB Micro, USB Type A, USB Type B, USB Type C, LPT, RS232, PS/2, or a combination of the above, and Any required transmission specifications can be replaced according to the needs of the application. In addition, in the embodiment of the present invention, the full-view conference camera device (100) can transmit conference images, panoramic images, split images, focused images, presentation images, etc. screens to external terminals through the transmission module (211). External terminals, such as smart phones, tablets, computers, smart wearable devices, etc., can use the transmission module (211) to transmit control commands to the processing module (201) to control the full-view conference camera device (100 ) Operation, its compatible operating system can be but not limited to Linux, Windos, Android, Mac OS or iOS, so that it can have better system compatibility.

Please refer to Figures 4A, 4B, 6B, 6C and 6D. According to the content of the present invention, the present invention proposes a cloud conference system, including: a number of full-view conference devices, which are arranged in at least two different geographic areas, wherein each full-view conference device There are several camera units to shoot conference images; the audio-visual processing module crops or collages the conference images taken by the aforementioned camera unit, wherein the audio-visual processing module further includes a target tracking unit to track at least one target location (such as Participants, moderators, presenters); split list Meta, according to the target location, the conference image is divided into a multi-screen mode, or collaged into a panoramic mode (Panorama mode). Among them, the audio-visual processing module also includes a presentation mode (Presentation Mode), which divides the presenter from the conference image, and stitches the presentation electronic data or written data into the presentation image; and the transmission module is used to connect to the remote end of the cloud to Run cloud meetings

In view of the above, according to an application of the present invention, when an online meeting starts, the aforementioned several meeting participants, moderators, or presenters may actually be located in different locations. Therefore, in order to cope with an online meeting with multiple locations There are multiple sets of full-view conference camera devices (100) that can be deployed in the several locations to form a cloud conference system (200), and connect each terminal to the cloud network through the transmission module (211). When the meeting starts, the camera unit of the cloud meeting system (200) will process the images of the participants into panoramic images, focused images, or split images through the aforementioned audio-visual processing module (209). . In addition, when the meeting host or presenter starts to speak, the target tracking unit (209a) is used to automatically or manually define the face and body of the meeting participant or host (such as the shape of each pixel in the camera unit). The video processing module (209) can continuously adjust the PTZ mode to divide the unit according to the position of the participant or host of the current meeting in the photographing unit. (209c) According to the needs of the online meeting, the position of the host or presenter can be tracked in the up, down, left, and right directions, or zoom. Furthermore, when the agenda of the online meeting is needed, the briefing unit (209e) can be used to activate the briefing mode (500a), so that the host, the presenter or the participants you want to follow can be viewed from the meeting image in the photographing unit The conference images of the required background (for example, the whiteboard, projection screen, or flat wall of the conference room) are spliced into a presentation image. Please refer to Figure 6C, which is an embodiment after the presentation mode (500a) is activated, in which the host, whiteboard, panoramic images, or even electronic data can be meetings taken by different photography units from different meeting locations The image is cut or collaged by the audio-visual processing module (209) into a presentation image to achieve the purpose of increasing the communication efficiency of the online meeting and the sense of participation of the meeting participants.

The above description is a preferred embodiment of the present invention to achieve the first to fourth objectives. Those skilled in this field should be able to understand that it is used to illustrate the present invention rather than to limit the scope of the claimed patent rights of the present invention. The scope of its patent protection shall be determined by the scope of the attached patent application and its equivalent fields. Anyone familiar with the art in this field, without departing from the spirit or scope of this patent, makes changes or modifications that are equivalent changes or designs completed under the spirit of the present invention, and should be included in the following patent applications Within range.

200: Full-view conference photography installation

201: Processing Module

203: The first photography unit

205: The second photography unit

207: The third photography unit

209: Audio and Video Processing Module

209a: Audio-visual tracking unit

209c: segmentation unit

209e: Presentation mode unit

211: Transmission Module

Claims (10)

A full-view conference camera device, and its system architecture includes: Multiple photography units to shoot conference images; An audio-visual processing module that cuts or collages conference images taken by the plurality of photographing units, wherein the audio-visual processing module further includes: A target tracking unit that tracks at least one target location; and, A dividing unit, according to the at least one target location, dividing the conference image into a multi-picture mode, or collage into a panorama mode (Panorama mode); The audio-visual processing module further includes a presentation mode (Presentation Mode), which divides the presenter from the conference image and stitches it with the presentation data into a presentation image; During the presentation mode, the presenter is focused through the target tracking unit. The full-view conference camera device according to claim 1, wherein the multi-view mode includes a top-down mode, and the conference image is processed into a split image with upper and lower split screens through the split unit. The full-view conference camera device according to claim 1, wherein the multi-screen mode includes a grid mode, and the conference image is processed into more than 4 divided images side by side and overlapping top and bottom through the dividing unit . The full-view conference camera device according to claim 1, wherein the presentation mode (Presentation Mode) combines electronic data or written data or panoramic images or segmented images with the presentation image to be displayed. The full-view conference camera device according to claim 1, wherein the target tracking unit tracks the at least one target position through a PTZ mode. A cloud conference system, including: A plurality of full-view conference devices are arranged in at least two different geographic areas, and each of the full-view conference devices includes a plurality of photographing units for shooting conference images, an audio-visual processing module, and cutting or collaging the plurality of photographing units. A captured conference image, wherein the audio-visual processing module further includes: a target tracking unit that tracks at least one target location; a segmentation unit that divides the conference image into a multi-screen mode or collage according to the at least one target location A panorama mode (Panorama mode), in which the audio-visual processing module further includes a presentation mode, which divides the presenter from the conference image and stitches it with the presentation electronic data or written data into a presentation image; and The transmission module is used to connect to the remote end of the cloud to perform cloud meetings. The cloud conference system according to claim 6, wherein the target tracking unit tracks the at least one target location through a PTZ mode. The cloud conference system according to claim 6, wherein the presentation mode (Presentation Mode) combines a panoramic image or a segmented image with the presentation image and displays it. The cloud conference system according to claim 6, wherein the at least one target is located in a different conference location, and a plurality of full-view conference devices in different conference locations are connected to the cloud network through the transmission module. The cloud conference system according to claim 6, wherein during the presentation mode, the presenter is focused through the target tracking unit.

Images