TWM656375U - Playback device - Google Patents

Abstract

A playback device developed through TouchDesigner, Reaper and SoundGrid can be used to perform dome-style video and audio playback and real-time image and audio serial output. The playback device uses two computer hosts, one for video and one for sound, and uses the Open Sound Control (OSC) communication protocol to achieve linkage operation between the computer hosts. In playback mode, the video and audio are played synchronously using Linear Timecode (LTC); thereby, this creation can perform ultra-high resolution (4Kx4K or above) image and spatial audio (Ambisonic) processing separately through two computer hosts, one for video and one for sound, and can be flexibly configured.

Description

Playback device

This work is about a playback device, especially one that uses two computer hosts, one for video and one for sound, and uses the Open Sound Control (OSC) communication protocol to achieve linkage between the computer hosts, especially in playback mode, where the video and audio are played synchronously using Linear Time Code (LTC).

With the advent of the 5G era, the problem of insufficient bandwidth for audio and video transmission has been solved, prompting the emergence of new types of multimedia interactive applications. Among them, the high interactivity and high immersion provided by 360-degree panoramic images are deeply loved by consumers and are also used in large-scale events such as exhibitions and performances, thereby bringing a sense of presence that traditional flat images cannot match.

The immersive experience space currently on the market uses projection to achieve a 360-degree panoramic image. However, the disadvantage of existing technology is that it is not easy for one computer to output and control multiple (more than 9) projectors and process ultra-high resolution (more than 4Kx4K), and to play spatial audio (Ambisonic) images and audio simultaneously on another device, and it cannot be flexibly configured. Therefore, general users cannot meet the needs of users in actual use.

The main purpose of this creation is to overcome the above problems encountered by the knowledge technology and provide a display device that can perform ultra-high resolution (4Kx4K or above) image and spatial audio (Ambisonic) processing separately by using two computer hosts, one for image and the other for sound, and can be flexibly configured.

To achieve the above purpose, this invention is a playback device, which is used to perform dome-style video and audio playback and real-time image and audio serial output, which includes: an image host, which is a computer host, composed of an image source selection module, an image processing module and a monitoring module, wherein: the image source selection module has a mode switching control interface and a real-time transmission unit, the mode switching control interface is used to switch a manual video selection mode or a play list mode in a playback mode, the manual video selection mode The video mode includes a manual video file selection unit for manually selecting a video file and its format and then outputting the video file to the image processing module. The playlist mode includes a playlist editor and a playlist control interface connected to the playlist editor. The playlist editor is used to set the playlist content and output the playlist to the image processing module after the total number of playlists is edited. The playlist control interface uses an open sound control (Open Sound Control, OSC) communication protocol to connect with the monitoring module and a sound project opening module; and the real-time transmission unit is used to provide a real-time transmission method of a capture card control interface, an NDI input control interface, and a Syphon/Spout input control interface in a real-time image input mode to input the real-time image to the image processing module by a capture card input, NDI input or Syphon/Spout; the image processing module has a video playback unit, an image detail adjustment unit, an image input source selection unit, an image screen format selection unit, an image calculation unit, an image display master switch and an image mapping unit, which loads the image from the image source selection module through the video playback unit. The video file or the playlist is selected, and whether to use the timecode to drive the video is switched. After the image detail adjustment unit performs image processing of brightness, contrast and opacity, the image input source selection unit and the image screen format selection unit connected to the manual video file selection unit and the playlist control interface can be used to control the selection of the image input source and the selection of the image screen format according to the property settings of the selected list. The image operation unit then performs image sphere deformation operation and converts it into a mode that conforms to the Dome for playback. The image display master switch controls the screen switch process with linear changes (Fade in, fade out). out), will not turn off instantly, turn on instantly, and then read the image calibration file in .VWS format through the image mapping unit, give the designated corresponding screen to the corresponding projector window according to the number of projectors, and perform spherical curved surface image welding and playback; the monitoring module has a Timecode signal input/OSC setting interface, an image adjustment interface, a playback content monitoring screen unit, an image display switch interface, and a projector sequence setting interface , an external lighting system control interface, and an interface for opening the Mapping projection screen. The image adjustment interface provides three adjustment parameters: brightness, contrast, and opacity, and outputs these adjustment parameters to the image detail adjustment unit. The broadcast content monitoring screen unit loads the result of the image calculation unit's image sphere deformation calculation to display the broadcast image content in real time. The image display switch interface will output whether the image is displayed as broadcast on (Map On) or broadcast off (Map Off). Off) to the image display master switch, the projector sequence setting interface will output the setting of projector resolution and total number of projectors, select the calibration file (.vwf) path, and set the projector sequence and set the corresponding allocated screen to the image mapping unit according to the calibration result. The Mapping projection screen opening interface will output the window screen corresponding to the turned-on projector to the corresponding projector window. The external lighting system control interface uses OSC signals to transmit an external lighting illumination instruction to a lighting master switch of an external lighting device to control the lighting on/off; and a sound host is another computer host, which achieves linkage operation with the image host through the OSC communication protocol. In the playback mode, the video and audio are synchronized by linear time code (Linear Time Code). Timecode, LTC) for synchronous playback. The sound host includes the sound project opening module and a sound playback unit. The sound project opening module is used in the playback mode. After receiving the OSC signal from the playlist control interface, it will open the corresponding sound project file according to the video set in the playlist, reducing the error rate in the playback operation and eliminating the need to frequently switch between two computer hosts during the operation process. The sound playback unit is used to play audio and spatial audio encoding and decoding processing, output LTC synchronous image playback, and finally output audio to the on-site audio equipment.

In the above-mentioned embodiment of the present invention, the video file format is a high-resolution image in a fish-eye format or an equirectangular format.

In the above-mentioned embodiment of this creation, the manual video file selection unit is used to manually select video files, input file paths, manually switch video image formats (Fish-Eye/Equirectangular), switch whether to bind Timecode to synchronize audio and video playback, and turn on/off the video playback loop function.

In the above-mentioned embodiment of this creation, the playlist editor is used to perform playlist content settings and edit the total number of playlists before outputting the playlist. The playlist content settings are used to set the playlist name, playback mode (file, capture card input, NDI input, Syphon/Spout input), set the file path, and image format (Fish-Eye/Equirectangular). The total number of playlists is used to increase or decrease the total number of playlists and automatically change the display of the playlist operation interface. The playlist control interface provides the ability to directly specify the video file to be played when clicking the playlist, and to send an OSC signal to the sound host to specify the corresponding sound project after clicking.

In the above-mentioned embodiment of the present invention, the capture card control interface further includes a capture card setting interface for executing multi-track input to obtain 4K*4K images arranged up and down to conform to the full dome picture format through a dual input capture card setting interface.

In the above-mentioned embodiment of the present invention, the image processing module further includes a switching unit connected to the manually selected video file unit, which is used to switch between playing with bound timecode or playing without bound timecode, and output the switching result to the video playback unit.

In the above-mentioned embodiment of the present invention, the monitoring module further includes a performance monitoring unit for displaying the number of frames per second (FPS) and the percentage of usage of the graphics processing unit (GPU).

In the above-mentioned embodiment of the present invention, the Timecode signal input/OSC setting interface utilizes the OSC communication protocol to connect the image processing module, the sound host and the external lighting device, and is used to execute the Timecode input setting, the OSC output to the sound host setting, the OSC output to the sound project start module automatic start setting, the OSC output to the external lighting device control setting, and the OSC input setting.

In the above-mentioned embodiment of the present invention, the image adjustment interface is an editing switch that can control whether to display the adjustment parameters.

In the above-mentioned embodiment of this creation, the sound playback unit uses Reaper and SoundGrid to process multi-channel audio and spatial audio encoding and decoding in spherical surround sound (Ambisonic) format.

In the above-mentioned embodiment of the present invention, in addition to the playback mode, the sound playback unit can also accept analog audio input or use SoundGrid Network digital audio for real-time audio input.

In the above-mentioned embodiment of the present invention, the sound project opening module includes a control interface, and a sound project list editor. The control interface provides a button to open the sound project list editor window for executing OSC input settings. The sound project list editor is used to generate a sound project list after editing and editing the total number of lists. The editor is used to set the list item name and specify the sound project file path. The total number of lists is used to increase or decrease the total number of lists.

In the above-mentioned embodiment of the present invention, the video playback unit will output an OSC signal to the sound host when the playback, stop playback, and return to the starting point are triggered, so that the sound project can be played, stopped, and returned to the starting point synchronously.

Please refer to "Figure 1 to Figure 2D", which are the block diagrams of the playback device of this creation, the block diagram of the image source selection module in the image host of this creation, the block diagram of the image processing module in the image host of this creation, the block diagram of the monitoring module in the image host of this creation, and the block diagram of the sound host of this creation. As shown in the figure: This creation is a playback device developed through TouchDesigner, Reaper and SoundGrid, which can be used to perform dome-style audio and video playback and real-time image and real-time audio serial output. The playback device uses two computer hosts, divided into an image host 1 and an audio host 2, and uses the Open Sound Control (OSC) communication protocol to achieve linkage operation between the computer hosts. In the playback mode of this device, the video and audio are played synchronously using the Linear Time Code (LTC).

The above-mentioned image host 1 is composed of an image source selection module 11, an image processing module 12 and a monitoring module 13. Among them:

The image source selection module 11 has a mode switching control interface 111 and a real-time transmission unit 112. The mode switching control interface 111 is used to switch a manual video selection mode 1111 or a play list mode 1112 in a playback mode. The manual video selection mode 1111 includes a manual video file selection unit 11111, which is used to manually select a video file and its format and then output the video file to the image processing module 12. The play list mode 1112 includes a play list editor 11121 and a play list control interface 11122 connected to the play list editor 11121. The playlist editor 11121 is used to set the playlist content and output the playlist to the image processing module 12 after the total number of playlists is edited. The playlist control interface 11122 uses the OSC communication protocol to connect to the monitoring module 13 and a sound project opening module 21; and the real-time transmission unit 112 is used to provide a real-time transmission method in a real-time image input mode, providing a capture card control interface 1121, an NDI input control interface 1122, and a Syphon/Spout input control interface 1123 to input the real-time image to the image processing module 12 by a capture card input, NDI input or Syphon/Spout.

The image processing module 12 has a video playback unit 122, an image detail adjustment unit 123, an image input source selection unit 124, an image screen format selection unit 125, an image calculation unit 126, an image display master switch 127 and an image mapping unit 128. The video playback unit 122 loads the video file or the play list from the image source selection module 11, and switches whether to use the time code to drive the video. After the image detail adjustment unit 123 performs image processing of brightness, contrast and opacity, the image input source selection unit connected to the manual video file selection unit 11111 and the play list control interface 11122 is used. 124 and the image screen format selection unit 125 can respectively control the selection of the image input source and the image screen format according to the property settings of the selected list, and then use the image calculation unit 126 to perform image spherical deformation calculation and convert it into a mode that conforms to the Dome for playback. The image display master switch 127 controls the screen switch process to have a linear change (Fade in, fade out), which will not be turned off or turned on instantly. Then, the image mapping unit 128 reads the image calibration file in the .VWS format, and gives the designated corresponding screen to the corresponding projector window 3 according to the number of projectors, so as to perform spherical curved surface image welding and playback.

The monitoring module 13 has a Timecode signal input/OSC setting interface 131, an image adjustment interface 132, a broadcast content monitoring screen unit 133, an image display switch interface 134, a projector sequence setting interface 135, an external lighting system control interface 136, and a Mapping projection screen interface 137. The image adjustment interface 132 is an editing switch that provides three adjustment parameters of brightness, contrast and opacity, and can control whether to display these adjustment parameters and output these adjustment parameters to the image detail adjustment unit 123. The broadcast content monitoring screen unit 133 loads the result of the image sphere deformation calculation by the image calculation unit 126 to display the broadcast image content in real time. The image display switch interface 134 will output whether the control image is displayed as Map On or Map Off to the image display master switch 127. The projector sequence setting interface 135 will output the setting of the projector resolution and the total number of projectors, the selection of the calibration file (.vwf) path, and the setting of the projector sequence and the setting of the corresponding allocated screen according to the calibration result to the image mapping unit 128. The Mapping projection screen opening interface 137 will output the window screen corresponding to the opening projector to the corresponding projector window 3. The external lighting system control interface 136 uses the OSC signal to transmit an external lighting on/off instruction to a lighting master switch 41 of an external lighting device 4 to control the lighting on/off.

The sound host 2 includes the sound project opening module 21 and a sound playback unit 22. The sound project opening module 21 is developed using TouchDesigner and is used in the playback mode. After receiving the OSC signal from the playlist control interface 11122, the sound project opening module 21 will open the corresponding Reaper sound project file according to the video set in the playlist, thereby reducing the error rate in the playback operation and eliminating the need to frequently switch between two computer hosts during the operation process. The sound playback unit 22 is used to play audio. It uses Reaper and SoundGrid to process multi-channel (e.g. 8-channel) audio systems and spatial audio encoding and decoding such as spherical surround sound (Ambisonic) formats, outputs LTC synchronized image playback, and finally outputs audio to the on-site audio equipment. In addition to the playback mode, it can also accept analog audio input or use SoundGrid Network digital audio for real-time audio input. In this way, a new playback device is constructed by the above-disclosed device.

In one of the preferred specific embodiments of this invention, the image host 1 is developed using TouchDesigner. In the playback mode, it can receive high-resolution images in fish-eye format or equirectangular format and convert them into a mode that conforms to Dome for playback. By reading the calibration file in .VWS format, the spherical curved surface images of 9 projectors can be welded and played.

In a preferred specific embodiment of the present invention, the manual video file selection unit 11111 is used to manually select a video file, input the file path, manually switch the video image format (Fish-Eye/Equirectangular), switch whether to bind the Timecode to synchronize audio and video playback, and turn on/off the video playback loop function.

In a preferred embodiment of the present invention, the playlist editor 11121 is used to set the playlist content and edit the total number of playlists before outputting the playlist. The playlist content setting is used to set the playlist name, playback mode (file, capture card input, NDI input, Syphon/Spout input), file path, and image format (Fish-Eye/Equirectangular). The total number of playlists editing is used to increase or decrease the total number of playlists and automatically change the display of the playlist operation interface.

In a preferred specific embodiment of the present invention, the playlist control interface 11122 provides a direct way to specify the video file to be played when clicking the playlist, and sends an OSC signal to the sound host 2 to specify the corresponding sound project to be opened after clicking.

In a preferred embodiment of the present invention, the capture card control interface 1121 further includes a dual input capture card setting interface 11211 for executing multi-track input to obtain 4K*4K images arranged up and down to conform to the full dome picture format.

In a preferred embodiment of the present invention, the image processing module 12 further includes a switching unit 121 connected to the manually selected video file unit, which is used to switch between playing with or without a Timecode, and output the switching result to the video playback unit 122.

In a preferred specific embodiment of the present invention, the video playback unit 122 will output an OSC signal to the sound host 2 when the functions such as play, stop, and return to the starting point are triggered, so that the sound project can be played, stopped, and returned to the starting point synchronously.

In a preferred embodiment of the present invention, the monitoring module 13 further includes a performance monitoring unit 138 for displaying the number of frames per second (FPS) and the percentage of graphics processing unit (GPU) usage.

In a preferred specific embodiment of the present invention, the Timecode signal input/OSC setting interface 131 uses the OSC communication protocol to connect the image processing module 12, the sound host 2 and the external lighting device 4, and is used to execute Timecode input setting, OSC output to the sound host setting, OSC output to the sound project start module automatic start setting, OSC output to the external lighting device control setting, and OSC input setting.

In a preferred embodiment of the present invention, the sound project opening module 21 includes a control interface 211 and a sound project list editor 212. The control interface 211 provides a window opening button for the sound project list editor 212 for executing OSC input settings. The sound project list editor 212 is used to generate a sound project list after editing and list total editing. The editing is used to set the list item name and specify the sound project file path. The list total editing is used to increase or decrease the list total.

In summary, this creation is a playback device that can effectively improve various shortcomings of usage. It uses two computer hosts, which are divided into an image host and a sound host, and uses the Open Sound Control (OSC) communication protocol to achieve linkage operation between the computer hosts. In playback mode, the video and audio of this device are played synchronously through Linear Timecode (LTC); thereby, this creation can perform ultra-high resolution (4Kx4K or more) image and spatial audio (Ambisonic) processing separately through two computer hosts divided into an image host and a sound host, and can be flexibly configured, thereby making the creation of this creation more advanced, more practical, and more in line with the needs of users. It has indeed met the requirements for a new patent application, and a patent application is filed in accordance with the law.

However, the above is only the best implementation example of this creation, and it should not be used to limit the scope of implementation of this creation; therefore, all simple equivalent changes and modifications made according to the scope of the patent application of this creation and the content of the new specification should still fall within the scope of the patent of this creation.

1: Video host 11: Video source selection module 111: Mode switching control interface 1111: Manual video selection mode 11111: Manual video file selection unit 1112: Play list mode 11121: Play list editor 11122: Play list control interface 112: Real-time transmission unit 1121: Capture card control interface 11211: Dual input capture card setting interface 1122: NDI input control interface 1123: Syphon/Spout input control interface 12: Video processing module 121: Switching unit 122: Video playback unit 123: Image detail adjustment unit 124: Video input source selection unit 125: Image screen format selection unit 126: Image calculation unit 127: Image display master switch 128: Image mapping unit 13: Monitoring module 131: Timecode signal input/OSC setting interface 132: Image adjustment interface 133: Playback content monitoring screen unit 134: Image display switch interface 135: Projector sequence setting interface 136: External lighting system control interface 137: Open Mapping projection screen interface 138: Performance monitoring unit 2: Sound host 21: Sound project opening module 211: Control interface 212: Sound project list editor 22: Sound playback unit 3: Projector window 4: External lighting device 41: Lighting master switch

Figure 1 is a block diagram of the creative playback device. Figure 2A is a block diagram of the image source selection module in the creative video host. Figure 2B is a block diagram of the image processing module in the creative video host. Figure 2C is a block diagram of the monitoring module in the creative video host. Figure 2D is a block diagram of the creative sound host.

1: Video host

11: Image source selection module

12: Image processing module

13: Monitoring module

2: Sound host

21: Sound project opening module

22: Sound playback unit

3: Projector window

4: External lighting device

Claims (13)

A playback device is used to perform dome-style video and audio playback and real-time image and audio serial output, which includes: An image host, which is a computer host, composed of an image source selection module, an image processing module and a monitoring module, wherein: The image source selection module has a mode switching control interface and a real-time transmission unit. The mode switching control interface is used to switch between a manual video selection mode and a play list mode in a playback mode. The manual video selection mode includes a manual video file selection unit, which is used to manually select a video file and its format and then output the video file to the image processing module. The play list mode includes a play list editor and a play list control interface connected to the play list editor. The play list editor is used to set the play list content and output the play list to the image processing module after the total number of the play list is edited. The play list control interface uses an open sound control (Open Sound Control, OSC) communication protocol to connect with the monitoring module and a sound project opening module; and the real-time transmission unit is used to provide a real-time transmission method in a real-time image input mode, providing a choice of a capture card control interface, an NDI input control interface, and a Syphon/Spout input control interface to input the real-time image to the image processing module via a capture card input, NDI input or Syphon/Spout; The image processing module has a video playback unit, an image detail adjustment unit, an image input source selection unit, an image frame format selection unit, an image calculation unit, an image display master switch and an image mapping unit. The video playback unit loads the video file or the play list from the image source selection module and switches whether to use the time code to drive the video. The image detail adjustment unit adjusts the brightness and contrast. After image processing of rast and opacity, the image input source selection unit and the image screen format selection unit connected to the manual video file selection unit and the play list control interface can be used to control the selection of the image input source and the image screen format according to the property settings of the selected list, and then the image calculation unit is used to perform image sphere deformation calculation and convert it into a mode that conforms to the Dome for playback. The image display master switch controls the screen switch process with linear changes (Fade in, fade out), which will not be turned off or turned on instantly. Then, the image mapping unit reads the image calibration file in .VWS format, and gives the designated corresponding screen to the corresponding projector window according to the number of projectors, so as to perform spherical curved surface image welding and playback; The monitoring module has a Timecode signal input/OSC setting interface, an image adjustment interface, a broadcast content monitoring screen unit, an image display switch interface, a projector sequence setting interface, an external lighting system control interface, and an interface for opening the Mapping projection screen. The image adjustment interface provides three adjustment parameters: brightness, contrast, and opacity, and outputs these adjustment parameters to the image detail adjustment unit. The broadcast content monitoring screen unit loads the result of the image calculation unit's image sphere deformation calculation to display the broadcast image content in real time. The image display switch interface outputs whether the control image is displayed as broadcast on (Map On) or broadcast off (Map Off) to the image display master switch, the projector sequence setting interface will output the projector resolution and total number of projectors, select the calibration file (.vwf) path, and set the projector sequence and set the corresponding allocated screen according to the calibration result to the image mapping unit, the Mapping projection screen opening interface will output the window screen corresponding to the turned-on projector to the corresponding projector window, and the external lighting system control interface uses the OSC signal to transmit the external lighting illumination command to a lighting master switch of an external lighting device to control the lighting on/off; and A sound host is another computer host, which uses the OSC communication protocol to achieve linkage operation with the video host. In the playback mode, the video and audio are played synchronously by linear time code (LTC). The sound host includes the sound project opening module and a sound playback unit. The sound project opening module is used in the playback mode. After receiving the OSC signal from the play list control interface, it will open the corresponding sound project file according to the video set in the play list, reducing the error rate in the playback operation and eliminating the need to frequently switch between two computer hosts during the operation process. The sound playback unit is used to play audio and spatial audio encoding and decoding processing, output LTC synchronous video playback, and finally output audio to the on-site audio equipment. According to the playback device described in item 1 of the patent application scope, the video file format is a high-resolution image in a fish-eye format or an equirectangular format. According to the playback device described in item 1 of the patent application scope, the manual video file selection unit is used to manually select video files, input file paths, manually switch video picture formats (Fish-Eye/Equirectangular), switch whether to bind Timecode to synchronize audio and video playback, and turn on/off the video playback loop function. According to the playback device described in item 1 of the patent application scope, the playlist editor is used to output the playlist after performing the playlist content setting and the playlist total editing. The playlist content setting is used to execute the setting of the playlist name, the playback mode (file, capture card input, NDI input, Syphon/Spout input), the file path, and the image format (Fish-Eye/Equirectangular). The total number of the list is used to increase or decrease the total number of the list and automatically change the display of the list operation interface at the same time. The playlist control interface provides the function of directly specifying the video file to be played when clicking the playlist, and sending an OSC signal to the sound host to specify the corresponding sound project to be opened after clicking. According to the playback device described in item 1 of the patent application scope, the capture card control interface further includes a dual input capture card setting interface for executing multi-track input to obtain 4K*4K images arranged up and down to comply with the full dome picture format. According to the playback device described in item 1 of the patent application scope, the image processing module further includes a switching unit connected to the manually selected video file unit, which is used to switch between Timecode-bound playback or unbound Timecode playback, and output the switching result to the video playback unit. According to the display device described in item 1 of the patent application, the monitoring module further includes a performance monitoring unit for displaying the number of frames per second (FPS) and the percentage of usage of the graphics processing unit (GPU). According to the playback device described in item 1 of the patent application scope, the Timecode signal input/OSC setting interface uses the OSC communication protocol to connect the image processing module, the sound host and the external lighting device, and is used to execute Timecode input settings, OSC output to the sound host settings, OSC output to the sound project start module automatic start settings, OSC output to the external lighting device control settings, and OSC input settings. According to the playback device described in item 1 of the patent application scope, the image adjustment interface is an editing switch that can control whether to display the adjustment parameters. According to the playback device described in item 1 of the patent application scope, the sound playback unit uses Reaper and SoundGrid to process multi-channel audio and spatial audio encoding and decoding in ambisonic format. According to the playback device described in item 1 of the patent application scope, the sound playback unit, in addition to the playback mode, can also accept analog audio input or use SoundGrid Network digital audio for real-time audio input. According to the playback device described in item 1 of the patent application scope, the sound project opening module includes a control interface and a sound project list editor. The control interface provides a button for opening the sound project list editor window for executing OSC input settings. The sound project list editor is used to generate a sound project list after editing and editing the total number of lists. The editor is used to set the list item name and specify the sound project file path. The total number of list editing is used to increase or decrease the total number of lists. According to the playback device described in item 1 of the patent application scope, the video playback unit outputs an OSC signal to the sound host when the playback, stop playback, and return to the starting point are triggered, so that the sound project is played, stopped, and returned to the starting point synchronously.

Images