JP2006211524A - Video display apparatus, control method of video processing apparatus, program, and storage medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To easily know contents of camera control that may occur next, when reproducing and displaying a video clip. <P>SOLUTION: Prior to the video display of a video file, camera control information contained in the video file is extracted, a video image of the video file is displayed and based on the extracted camera control information, camera control contents relating to video images of time bands with no display are displayed. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a video display device that displays a video file received from a camera server connected via a network, a control method for the video processing device, a program, and a storage medium.

  2. Description of the Related Art There is known a technique for reproducing and using video clips when video data (video clips) created by reflecting control information of a live video source such as a camera device is reproduced. The existing technology related to this technology will be described below.

<Live video communication system>
A technology for instructing camera settings for camera shooting and camera operation, etc., has been established and products are being sold while delivering live video using a communication infrastructure such as the Internet. For example, Canon's video distribution system WebView / Livescope, the company's live network camera server VB101, network camera VB-C10, and the like.

  In the video distribution system of Canon Inc., in addition to video distribution, camera control such as pan, tilt, zoom, and backlight correction can be provided via a network. In addition, an access control function is provided, and camera control and video distribution can be restricted according to the access authority of the user.

  Furthermore, it is possible to limit the area to be imaged by camera control. For example, the privileged user can use all of the zoom functions provided in the camera, but the normal user is limited so that only a part of the zoom function (for example, the tele end cannot be used up) can be used. The same applies to the pan function and tilt function.

<Third generation mobile phone technology>
A third generation (3G) mobile phone service is provided as a mobile phone service having higher radio wave use efficiency and communication band than conventional mobile phone services. For example, NTT Docomo's third generation (3G) mobile phone service FOMA and KDDI's cdma2000 1x.

  The third generation (3G) mobile phone can perform data communication such as Internet access while making a telephone call. For example, the FOMA service provided by NTT Docomo provides a connection form called multi-access, which makes it possible to make a telephone call while performing data communication such as web browsing.

  Furthermore, in the third-generation mobile phone terminal, the processing capability of the terminal itself has been strengthened, so that work that has been performed on a PC (personal computer) or the like can be processed by the mobile phone terminal. For example, mobile phone terminals equipped with functions such as mail, web browsing and video transmission / reception have been provided. In the third-generation mobile phone service, video distribution service is also provided, and services such as i-motion and M-stage Visual from Docomo, or ezmovie from KDDI are provided.

<MPEG-4 codec>
High compression coding efficiency covering a wide bit rate from several tens of kbps to several tens of Mbps in response to the spread of video transmission / reception terminals from portable information terminals connected to mobile communication networks to PCs connected to broadband Internet, and MPEG-4 was established in 1999 by ISO as a moving image compression encoding system having strong resistance to transmission path errors such as radio and the Internet.

  In addition, a video distribution service using MPEG-4 is provided for personal information terminals (PDAs) and mobile phone terminals. For example, the third generation (3G) mobile phone service of NTT Docomo or KDDI provides a service for transmitting and receiving video between mobile phone terminals (visual terminals) using MPEG-4.

<MPEG-4 clip technology for mobile phones>
A technique for displaying a video clip (file) on a mobile phone terminal is provided. For example, there are i-motion and M-stage visual from NTT Docomo, or ezmovie from KDDI.

  In these services, video data (video clips or video files) compressed and encoded by the MPEG-4 codec is stored on the server and downloaded from the server using the data communication function built in the mobile phone terminal. Similarly, video is displayed on the screen of the mobile phone terminal using a decoder built in the mobile phone terminal.

  In addition, the data format of these video clips is widely spread on the Internet and PCs such as Microsoft's Advanced Streaming Format (ASF) format and the ISO standard MP4 format (ISO / IEC 14496-1 Amd1 MPEG-4 system Version 2). It conforms to the format.

  Also, in these services, the upper limit of the video clip is determined. For example, in the case of KDDI ezmovie, the upper limit is 240 kbytes.

<Link technology to video clips and command association technology>
In Microsoft's ASF (Advanced Streaming Format) format, Apple's QuickTime File Format, and the like, hyperlink functions such as URLs can be associated with video clips.

  For example, in ASF, “Script Command Object” can be defined, and link information set to synchronize with the timeline at the time of ASF file playback can be listed in this object. Furthermore, in the ASF, command information such as script as well as link information can be described as the name of Script Command Object.

  The ezdi specification of KDDI also has a function of adding a text telop (caption) with a hyperlink function to a video clip. As this telop description language, KDDI's STML (abbreviation of Synchronous Telop Mark-up Language) is used. With this function, the user can associate voice calls, e-mail transmissions, homepage links, and the like with video clips. The same applies to 3GPP (The 3rd Generation Partnership Project) video clip specification 3gp.

<Technology for incorporating camera control sequences into video clips>
Patent Document 1 discloses a method of incorporating camera control information for controlling a camera clip into a video clip generated using a camera server arranged on a network as a video source.

  Here, a method is shown in which the camera control information sequence at the time when the video clip is generated is incorporated into the video clip so as to be synchronized with the video content along the time series of the video clip. Further, a method is shown in which the camera control information sequence is applied to a camera server at the time of video clip playback, and a live video controlled in accordance with the camera control information sequence is acquired.

<Digital STB technology>
As a device for receiving a digital TV broadcast on a CATV, an Internet TV, or the like, a device called a digital STB (Set Top Box) has been developed. These devices have a function of receiving streaming video and video clips (files) and displaying playback video on a TV screen. In many cases, standard video codecs such as MPEG-2 and MPEG-4 are used as these video codecs. As a streaming communication procedure, RTSP (Real Time Streaming Protocol; RFC2326), RTP (Real-time Transport Protocol; RFC1889), or the like is used.

Furthermore, organizations such as ISMA (Internet Streaming Media Alliance) have proposed specifications that comprehensively summarize these individual rules for the purpose of improving the interoperability between devices. For example, in ISMA, MPEG-4, RTSP, and RTP are basically used, and MPEG-4 Simple Visual Profile, QCIF (176 × 144), 15 fps, and 64 Kbps max are defined as ISMA Profile 0. Similarly, MPEG-4 Advanced Simple Profile (ASP) or Simple Visual Profile, CIF (352 × 288), 30 fps, 1.5 Mbps max is defined as ISMA Profile 1.
JP 2004-304651 A

  However, when providing video information from a vast number of camera servers that can be controlled on the Internet as video clips, camera control information (pan, tilt, zoom, control right, etc.) attached to the video information Information) could not be used properly. Alternatively, even if it can be used, it has been difficult to use the camera control information included therein to improve the convenience of video clip playback.

  Conventionally, it has been possible to detect breaks such as scene division by analyzing the content of video content at the image level. However, it requires a lot of computing power and is required for mobile phone terminals and STB (SetTopBox). It has been difficult to realize the above in terms of computing resources or power consumption.

  In addition, because of the explosive increase in video clips distributed on the Internet and the appearance of dynamically generated video clips such as network cameras, the supply of video clips increases, and simply improving hardware performance solves the problem. It has not reached.

  The present invention is intended to solve the above-described problems, and a video display device, a video processing device control method, a program, and a video processing device capable of easily knowing the next camera control content at the time of video clip reproduction display. And a storage medium.

  In order to achieve the above object, a video display device of the present invention is a video display device that displays a video file received from a camera server connected via a network, and prior to video display of the video file, Information extracting means for extracting camera control information contained in the video file; and displaying the video of the video file and relating to video in a time zone not displayed based on the camera control information extracted by the information extracting means. Display means for displaying the contents of camera control.

  The video display device control method of the present invention is a video display device control method for displaying a video file received from a camera server connected via a network, prior to video display of the video file, An information extraction step for extracting camera control information included in the video file, and a video in the time zone that is not displayed based on the camera control information extracted by the information extraction step while displaying the video of the video file. And a display step for displaying camera control content.

  The program of the present invention is a program for executing a control method of a video display device that displays a video file received from a camera server connected via a network, prior to video display of the video file, An information extraction step for extracting camera control information included in the video file; and a video of the video file is displayed, and a video in a time zone that is not displayed is displayed based on the camera control information extracted by the information extraction step. And a display step for displaying related camera control contents.

  The storage medium of the present invention is a storage medium storing a program code for executing a control method of a video display device that displays a video file received from a camera server connected via a network, the video Prior to video display of the file, an information extraction code for extracting camera control information included in the video file and a video of the video file are displayed and displayed based on the camera control information extracted by the information extraction code. And a display code for displaying camera control contents related to the video in the non-time zone.

  According to the present invention, it is possible to easily know the camera control content that will occur next when the video clip is reproduced and displayed. In addition, connection to the camera server according to the contents of the video clip is facilitated.

  Embodiments of the present invention will be described below with reference to the drawings.

(First embodiment)
In the first embodiment, an example will be described in which a video clip with camera control information generated from live video of a large number of camera servers arranged on a network is played back.

  In particular, during video clip playback, the video clip is prefetched, the change in the camera control state that becomes the video break is detected, the change in the camera control state is predicted, the input from the user is accepted, and the user input If there is (button click etc.), the feature is that the video of the forecast point is provided by fast-forwarding the video clip.

  FIG. 1 is a diagram showing a system configuration of the present embodiment, in which 101 and 102 are camera servers using a live video communication system, and 200 is a viewer installed on a PC or the like connected to a network. The camera server and viewer are each connected to a network, and a request is sent from the viewer to the camera server via the network. When the request is accepted, video data is delivered from the camera server to the viewer, and the camera video is displayed on the viewer. It becomes possible. In addition, a camera control command is sent from the viewer to the camera server, and operations such as zooming, panning, and tilting of the camera are possible. Further, a relay server 300 may be placed on the network to relay communication between the viewer and the camera server.

  Further, reference numeral 400 denotes a video conversion server to which the present invention can be applied. The video data provided by the camera server is converted for a mobile phone terminal and then relayed to the mobile phone.

  Reference numeral 500 denotes a gateway that mediates between the network of the present embodiment and the mobile phone network, and reference numerals 601 and 602 denote mobile phone terminals equipped with a viewer to which the present invention can be applied. Via the gateway 500, the devices connected to the network of the present embodiment and the mobile phone terminals 601 and 602 can communicate. Note that the viewer program on the mobile phone terminal is typically installed at the time of shipment from the factory, but is downloaded and executed at the time of execution (when used) like a Java (registered trademark) program. Also good.

  The network shown in the figure may be an intranet operated in a company or organization, or may be the Internet that widely connects the world. In addition, the relay server and the video conversion server are typically arranged in an Internet exchange (IX) or a data center (IDC: Internet Data Center) and designed to reduce a communication load.

  FIG. 2 shows an example of a hardware configuration for operating the video conversion server 400. The computer for the server, more specifically, a storage device for storing programs and data, and a network I / O for connection to a network. F2001, CPU2002 which performs various processes by a program, etc. consist of. The storage device includes a RAM 2003 serving as a main storage device, a secondary storage device 2004 including a flash memory and an HD device, and an FD device 2005 for loading a program from a medium.

  Further, although not shown, there may be an input / output device for performing setting or the like. Specifically, a display device to which a display is connected, a controller such as a keyboard and a mouse, and the like.

  FIG. 3 shows an example of the hardware configuration of the camera server 102. The camera device 3001 and the computer that actually take an image, more specifically, a storage device that stores programs and data, and a video data fetcher. The video capture board 3002, a serial I / F 3003 for sending commands to the camera device 3001, a network I / F 3004 for connecting to a network, a CPU 3005 for executing various processes by programs, and the like. The storage device includes a RAM 3006 serving as a main storage device, a secondary storage device 3007 including a flash memory and an HD device, and an FD device 3008 for loading a program from a medium.

  Further, although not shown, there may be an input / output device for performing setting or the like. Specifically, a display device to which a display is connected, a controller such as a keyboard and a mouse, and the like.

  Further, although not shown, the camera server 102 may be configured by a server built-in network camera in which the camera device 3001 and a computer are integrated.

  FIG. 4 shows an example of a hardware configuration for operating the viewer 200, and describes a case where the viewer is operated using a mobile phone terminal. More specifically, a storage device 4001 that stores programs and data, a wireless communication I / F 4002 for connection to a mobile phone network, a CPU 4003 that executes various processes according to the program, and a display controller 4005 that performs display control in the display device 4004 And a peripheral controller 4006 for controlling peripheral devices. The storage device 4001 includes a RAM that is a main storage device, a flash memory, and the like. Peripheral devices include input devices such as buttons 4007 and switches 4008 arranged on a mobile phone terminal, display output devices such as a display device 4004, and audio input / output devices 4009 such as a microphone and a speaker.

  FIG. 5 is a diagram schematically showing the components of the program, and the camera servers 101 and 102 include two modules, a camera control server 5001 that controls the camera and a video server 5002 that controls video distribution.

  Similarly, in the viewer 200 operating on the mobile phone terminal, a camera control input unit 5003 corresponding to the issuance of camera control commands and camera state notification, a video display unit 5004 responsible for displaying video clips, and a plurality of videos A video clip management unit 5005 for identifying and managing clips is included.

  Further, the video conversion server 400 interprets a camera control command sequence (hereinafter referred to as a PTZ sequence), acquires a video from the camera control unit 5006 that issues a camera control command to the camera server, and the mobile phone. Modules (video acquisition unit 5007, video conversion unit 5008, video transmission unit 5009) for converting to a phone and editing it into a video clip for a mobile phone network are included.

  FIG. 6 is a diagram showing the flow of the operation of the video clip viewer for reproducing and displaying the video clip on the mobile phone.

  In step S601, first, the identifier of the video conversion server 400 is obtained. In this case, the user may directly input an identifier by a key operation, or may select an identifier included in a mail or a web page. The identifier in this case is typically a URL that identifies the video conversion server 400.

  Subsequently, in step S602, the identifier of the camera server to which the video conversion server 400 is to be connected, the PTZ sequence for camera control instructed to the camera server, and the user identification information for access control such as the user identifier and password are obtained. .

  Here, the camera server identifier and the PTZ sequence may be input by the user so as to constitute a camera control command by direct key operation, or the camera server identifier and the PTZ sequence included in the mail or web page are selected. Also good. The PTZ sequence may be empty. In this case, it means a current video clip of the camera server without camera control. In addition, the user identification information is normally input by a user directly through a key operation, but may be empty. In this case, it means connection to the camera server as a normal user.

  In step S603, the video transmission server 400 connects to the video transmission unit of the video conversion server 400 based on the obtained identifier of the video conversion server 400 via the gateway 500. In step S604, the cellular phone viewer 200 requests the video conversion server 400 for a video clip. This request includes designation of a method for creating a requested video clip, and is typically transmitted according to the HTTP protocol.

At this time, if the PTZ sequence is obtained, the PTZ sequence is transmitted to the video conversion server 400. This request and the transmission of the PTZ sequence may be incorporated into the URL by the HTTP connection GET method, or may be transmitted by the HTTP connection POST method. Here, a case where transmission is performed by the POST method will be described. For example, it is as follows. In practice, URL encoding is applied. However, in the following, there is a part where URL encoding is not applied for convenience of explanation.
POST /getvideoclip/HTTP/1.1
Host: 202.28.30.208:8080
User-Agent: MozillaPhone / 2.0 C2101V (c100)
Pragma: no-cache
videoencodeparam = QCIF: fps15.0: bps64000: intraframe5: me8
cameraservers = webview: // vb101_4. x-zone. canon. co. jp: 34560 + 34561 + 34562
PTZ = HZ15_30S3_40S4_40P-10_20
moviesizemax = 240 kbytes
notifyto = mailto: riyousha3 @ mailserver. usersite. co. jp
userid = 331245
userpw = 1515294
However, the portion following “videoencodeparam =” indicates parameter information when the video conversion server 400 encodes the source video received from the camera server for the mobile phone. The part following “cameraservers =” designates the camera server to which the video conversion server 400 is connected. The part following “PTZ =” designates a PTZ sequence in which commands for camera control to be executed by the video conversion server 400 regarding the camera server are arranged. The part following “moviesizemax =” designates the maximum size of the video clip defined by the mobile phone terminal or the mobile phone network. Further, the part following “notifyto =” is a contact to be notified when creating a video clip. Typically, the mail address of the mobile phone terminal user is designated. The portions following “userid =” and “userpw =” are a user identifier and a password, respectively.

Among these, the components of the PTZ sequence have the following meanings. n is numerical data (positive and negative).
Pn pan (horizontal camera control) designation.
Tn tilt (vertical camera control) designation.
Zn Zoom designation.
Bn Backlight correction. ON / OFF for backlight compensation.
H Specify home position.
Sn Preset position designation. nth preset position.
Cn Camera server connection switching specification. The nth camera server.
Kn Specify camera switching within the camera server. nth camera.
_N Time elapsed specification. 0.1 second unit.

  In step S605, a response from the video conversion server 400 is waited for. In step S606, the response received from the video conversion server 400 is interpreted and displayed on the display device of the mobile phone terminal. For example, as shown in FIG.

  If the response from the video conversion server 400 indicates that the video cannot be generated immediately for some reason (S607, No), the process proceeds to step S609. On the other hand, if the content indicates that the video can be generated immediately (S607, Yes), the process proceeds to step S608.

  In step S609, a mail notification from the video conversion server 400 is awaited. The mail notification is based on, for example, SMTP (Simple Mail Transfer Protocol), but may be SMS Short Message Service (SMS). If a mail notification is received, the mail content is displayed as a response from the video conversion server 400, and the process proceeds to step S608. For example, as shown in FIG.

  In step S608, one of the video clips included in the response is selected, downloaded, and reproduced and displayed. For example, as shown in FIG. Although the download completion is awaited here, the reproduction display processing of the video clip may be started when video clip data that can be displayed / reproduced is ready without waiting for the completion of the download.

Among these, during the video clip display in the mobile phone viewer in step S608, the link information portion in the video clip is prefetched, and the camera control information incorporated as the link information is analyzed as shown in FIG. Then, the camera control content is predicted on the video display screen, and when the user presses the number button, it jumps to the video segment (video chapter) of the video clip corresponding to the link information of that number (video fast forward). . For example, when the number button “3” is pressed, the video clip playback is advanced by 42 seconds, and the video when the preset # 90 is selected is played back and displayed as camera control.
[1] 8 seconds later Pan right [2] 10 seconds later Zoom far [3] 42 seconds later Preset # 90
[4] After 50 seconds Camera # 2
[5] 83 seconds later Tilt down [6] 96 seconds later Preset # 5
In addition, when the mobile phone terminal user clicks (presses the selection button) while the video clip is displayed on the mobile phone viewer in step S608, the video data (video section or video segment) being displayed at the time of the click is displayed. Link information corresponding to the link information is extracted from the video clip, and the link destination information indicated by the link information is accessed using the browser function provided in the mobile phone terminal. In accessing link destination information, a dedicated viewer that enables camera control is typically activated, but depending on the setting of the mobile phone terminal, a set action may be performed. For example, it is a transmission of a mail with the accessed link information attached as a mail.

  FIG. 14 illustrates an operation flow of a live video viewer that communicates with the camera server to display a live video and perform camera control.

  In step S651, connection is made to the video server in accordance with the address and connection port information of the video server constituting the camera server instructed at the time of startup. Here, an operation program for performing processing after connection (actually, a thread or process is activated) is activated, and this program repeats step S661 until the end. In step S661, every time video data from the video server arrives, it is received and displayed.

  Further, in step S652, the main program connects to the camera control server in accordance with the camera control server address and connection port information that are also instructed at the time of activation.

  Thereafter, the main program receives an operation request from the user and continues to the main loop to be executed.

  First, in step S653, a user operation is received from a key button operation or the like. If this is related to camera control, a command is issued to the camera control server in step S654.

  If it is related to the video server, a command is issued to the video server in step S655. If the user operation is an operation for changing the viewer state (for example, a display size changing operation), the internal state is updated in step S656.

  If the user operation is finished, the programs related to the operation of the viewer are sequentially finished in step S657. When the processes of S654 to S657 are completed, the process returns to S653 and waits for a user operation input.

  A live video viewer that operates on a mobile phone terminal may be implemented as software provided at the time of shipment of the mobile phone terminal, but is a software that is downloaded and implemented from a network, such as a Java (registered trademark) program. There is also.

  FIG. 7 is a flowchart showing the operation of the camera control server in the camera server. First, the camera control server reads operation setting information of the camera control server from a specific file (a system database such as a registry depending on the OS) at the time of activation in step S701, and starts an operation based on the information. Here, a port for accepting a request from a viewer program as a client or a video conversion server is opened, and then a request acceptance state of step S702 is entered.

  If a request (connection request or operation command request) is accepted, the process exits S702, and if it is a connection request, it is determined whether or not connection is possible in step S703. If not, a connection rejection error code is returned, and the process returns to S702. If YES in step S704, a thread for receiving a command from the client is generated as a connection process in step S704, the client is registered, and the process returns to step S702.

  In the generated thread, a command is received from the corresponding client in step S707. If a command arrives, it is accepted and passed to the main program that operates the camera. The main program receives this in step S702, and proceeds to step S705 for the operation command, performs the camera operation according to the authority of the client to which the thread that issued the operation command is connected, and the result (operation succeeded). Is transmitted to the thread corresponding to the client that has received the camera operation request.

  The thread corresponding to the client sends the result back to the client in step S708. In the main program portion, in step S706, the state changed by the camera operation (for example, camera state information including pan / tilt / zoom values, presence / absence of prohibited area detection, etc.) is transmitted to all client-corresponding threads. . In step S709, the thread corresponding to each client notifies the client of the change in the camera control state. If the thread corresponding to the client receives a connection termination command from the client, it notifies the main program, and terminates its own thread in step S710.

  In handling the operation command, it is possible to require a camera operation right assignment request before issuing a specific operation command. This eliminates confusion in situations where multiple people require camera operation. In this case, first, a request command for acquiring the camera operation right is issued from the client. In response to this, the camera control server selects rejection / allocation / wait in order from the current assignment state of the camera control right and responds to the client. . The camera control right is deprived at a predetermined time or a shorter time until the client terminates the connection, and assigned to the next waiting person. The number of people waiting in the queue is still limited to a predetermined number (for example, 5 people), and further requests are rejected.

  The client issues an operation command only from when the camera control right is acquired until it is revoked. The camera control server accepts only operation commands from clients to which camera control rights are granted.

  In addition, with respect to connections from privileged users, the point that camera operation rights are preferentially assigned, and that operation commands are permitted even when camera control is performed on an imaging area including a prohibited area, etc. It is different from the connection from.

  FIG. 8 is a flowchart showing the operation of the video server in the camera server. The video server first reads the operation setting information of the video server from a specific file (a system database such as a registry depending on the OS) in step S801 at the time of start-up, and starts the operation based on the information. Here, a thread for acquiring, encoding, and accumulating video is generated (initially this thread is in a dormant state), and a port for receiving a request from a viewer program as a client or a video conversion server is opened. Subsequently, in step S802 Enter the request acceptance state.

  If a request (connection request or command request) is accepted, the process exits S802, and if it is a connection request, it is determined whether or not connection is possible in step S803. If not, a connection rejection error code is returned, and the process returns to S802.

  If yes, as a connection process in step S804, a session identifier for identifying a session for each client is generated, a thread for receiving a command from the client is generated, and the access right information of the client that issued the connection request The client is registered in accordance with the above, and the process returns to S802. At this time, if the request content is a connection to a live video and the thread for acquiring and encoding the video is in a dormant state, an operation start is instructed before returning to S802.

  The generated client-compatible thread accepts a command from the corresponding client in step S807. If a command arrives, it is received and transferred to the main program that performs video processing.

  The main program receives this in step S802, and proceeds to step S805 in response to the operation command, performs a setting change operation regarding acquisition, encoding, transmission, etc. of the video, and the result (a code indicating success or failure of the operation). ) To the thread corresponding to the client that received the command request.

  The thread corresponding to the client sends this result back to the client in step S808.

  In the main program part, in accordance with an instruction to start the operation from the acquisition and encoding thread from step S804, in step S806, video data is acquired using a video capture board at a preset time interval. Convert to compressed data. Further, this compressed data is transmitted to all client-compatible threads connected to the live video.

  In step S809, the thread corresponding to each client determines whether there is a next video frame transmission request from the client, and if there is a request, delivers the compressed data to the client.

  At this time, if a prohibited area is detected, the compressed data is not distributed because it corresponds to the prohibited area except for the connection of privileged users in light of the registered client information (detection of prohibited area). Notification). Then, the client-compatible thread connected to the live video receives the next video frame transmission request from the client (this is generally returned when the client receives the compressed video data). If it is, the video frame transmission request flag is set.

  If a connection termination command is received from the client, it is notified to the main program, and its thread is terminated in step S810.

  FIG. 10 shows an example of a display screen of a camera server setting program for setting setting values used by the camera server, that is, operation setting information read by the camera control server or video server in a specific file (system database such as a registry depending on the OS). In the figure, various parameters (described later) relating to a camera control server, a video server, moving image quality, connection restriction items, and the like can be set. When the OK button is pressed, the set value is written to a specific file or registry, and when canceled, the process ends without being written.

  FIG. 11 is a flowchart showing the operation of the setting program of the camera server in FIG. At the time of activation, the setting program first reads setting information from a specific file (system database such as a registry depending on the OS) that stores setting information regarding the camera control server and the video server in step S1101, and sets the setting information as internal data. Thereafter, the user's operation input is received and the loop to be executed is repeated.

  In step 1102, the user waits for an operation input. If there is an input, it receives it. Then, in step 1103, it is determined whether the input value is within an appropriate range. If not, an error message is output in step 1104. Output, return the value, and return to the user input wait S1102. If it is within the proper range, the internal data is updated, and the process returns to S1102. (Values that can be set here include the following items: TCP port number for camera control communication, COM (serial) port connected to the camera, shutter speed, presence / absence of log information related to camera control, and log file name, TCP port number for video-related communication, presence / absence of log information and log file name, frame rate that defines the time interval for capturing video, Q-Factor that determines compression quality, screen size of original data of compression, one The maximum connection time of the client viewer, the number of persons waiting for the control right related to the camera control, the control right holding occupation time of one viewer, the maximum number of connectable clients related to video and camera control, etc.).

  If the input from the user is an OK button, the process advances from step S1102 to step S1105, and the updated internal data is written to a specific file for storing setting information related to the camera control server and the video server, and the change is reflected in step S1106. A panel that asks if you want to restart the camera server. When restarting, the camera control server, the video server, etc. are restarted in step S1107, and the setting program is terminated in step S1108. If it is not restarted, the process directly proceeds from S1106 to S1108 and ends. If the user input in S1102 is a cancel button, the process proceeds directly from S1102 to S1108 and ends.

  FIG. 12 is a diagram schematically showing a rough flow of video data in the video conversion server. Source video data (Motion JPEG, QVGA size 320x240) transmitted from the camera server is received via the camera server communication stack of the video conversion server, passed to the JPEG decoder, and then set for the mobile phone. Passed to the MPEG-4 encoder, processed into video data for mobile phones (MPEG-4 simple profile, QCIF size 176x144, 64 Kbps), and then transferred as a video clip via a communication stack for mobile phone networks. Sent to the phone viewer.

  FIG. 13 is a flowchart showing the operation of the video conversion server. The video conversion server first reads the operation setting information of the video conversion server from a specific file (or a system database such as a registry depending on the OS) in step S1301 at the time of start-up, and starts the operation based on it. Here, a communication port that accepts a request from the mobile phone viewer program as a client is opened, and then the request acceptance state of step S1302 is entered.

  When a request (such as an HTTP request message) is received, the process exits S1302 and determines whether or not connection is possible in step S1303. If not, a connection rejection error code is returned, and the process returns to S1302. If YES in step S1304, as a connection process, a client-compatible thread for exchanging information with the client is generated, the client is registered, and the process returns to S1302.

  In the generated client corresponding thread, a request from the corresponding client is read in step S1311 and the content is analyzed. The request is typically passed to the video conversion server as an HTTP request. Note that a POST method may be used for the HTTP request, and a GET method may be used.

  In step S1312, from the request content, encoding parameter information (video conversion parameter), connection information to the camera server (source video information), PTZ sequence, maximum size of video clip (upper limit value of video clip), notification destination information (Notification destination address) and user identification information such as a user identifier and a password are extracted. These are indicated as values of “videoencodeparam =”, “cameraservers =”, “PTZ =”, “moviesizemax =”, “notifyto”, “userid =”, and “userpw =”, respectively.

  The video conversion parameters describe selection of a conversion codec, parameters to the codec, a data format for codec input / output, and the like. The source video information is, for example, communication attribute information such as a network address and a port number of a camera server that provides a live video. The notification destination address is typically a mail address that designates a user's mobile phone terminal.

  In step S1313, information indicating that “the video cannot be generated immediately and will be contacted in a while” will be returned as an HTTP response to the HTTP request. Next, in step S1314, the video acquisition unit is initialized according to the source video information and the user identification information. Specifically, it connects to the camera server that provides the source video and starts acquiring the source video.

  In step S1315, the video conversion unit is initialized according to the video conversion parameters. This video conversion unit includes an MPEG-4 encoder or the like. In step S1316, the video transmission unit is initialized. At this time, the video transmission unit is instructed to the video clip upper limit value and the notification destination address.

  Further, the process proceeds to step S1317, and the camera control unit performs the correlation so that each processing data is transferred from the video acquisition unit to the video conversion unit and from the video conversion unit to the video transmission unit. Camera control of the camera server is performed according to the sequence. In step S1318, post-processing is performed on the video acquisition unit, the video conversion unit, and the video transmission unit. Then, the process proceeds to step S1319, and the client-compatible thread is terminated.

  Next, the video acquisition unit 5007, the video conversion unit 5008, the video transmission unit 5009, and the camera control unit 5006 that function in the video conversion server of FIG.

  The video acquisition unit 5007 first connects to a camera server (hereinafter referred to as camera server 101) that provides a live video according to the source video information and user identification information received at the time of initialization. Then, video data is acquired from the camera server 101, a time stamp at the time of acquisition is added, and the video data is passed to the video conversion unit 5008. Since the camera server 101 in this embodiment provides video data in the Motion JPEG format, each JPEG data is given a time stamp. In addition, when a prohibited area detection is notified from the camera server, a prohibited area detection notification is passed to the video conversion unit 5008 instead of the video data.

  Next, the video conversion unit 5008 first sets the parameters for the codec received at initialization, the data format for codec input / output, and the like in the MPEG-4 encoder. Then, the source video data received from the video acquisition unit 5008 is adjusted to the data format and image size for codec input and then input to the MPEG-4 encoder, and the processing result is passed to the video transmission unit 5009. In the video conversion unit 5008 in this embodiment, source video data in JPEG format is preliminarily arranged in QCIF size and YUV411 format, and then input to the MPEG-4 codec, and the generated MPEG-4 data (I-frame or P -Frame) to the video transmission unit 5009. At this time, the time stamp given by the video acquisition unit 5007 is also input to the MPEG-4 codec.

  When the prohibited area detection is notified from the video acquisition unit 5007, a composite screen indicating that the video cannot be displayed because the camera control is limited is replaced with the source video data instead of the MPEG-4. Input to the codec.

  Next, video transmission section 5009 first secures a memory area according to the video clip upper limit value received at the time of initialization. Then, the mobile phone video data generated by the video converter 5008 is received and held in the secured memory area.

  Furthermore, when the video transmission unit 5009 receives the prohibited area detection notification, the video clip division point is determined in consideration of the utilization rate of the memory area. If it is determined as a division point, the header information conforming to the data format of the video clip for mobile phones is prepended, the video data held in the memory area is saved as a file, and the memory area is reused. To do. As a result, the video clip is divided and saved into a plurality of files.

  The video transmission unit 5009 receives camera control state information obtained by the video acquisition unit 5007 and holds it along the time axis. Then, a camera control sequence (PTZ sequence) corresponding to the camera control state information is generated from the camera control state information at a preset time period, and the support for starting the dedicated viewer using the PTZ sequence as a parameter is linked information. As a video clip. Typically, a PTZ sequence is formed from the difference between adjacent camera control state information. When the initial value of the PTZ sequence is designated, or when it matches the preset position, home position, etc., an absolute value designation is instructed.

  When the end of the PTZ sequence is notified from the camera control unit 5006, the remaining video data stored in the memory area is similarly stored as a file, and a plurality of video clips stored so far are stored. The notification information to the mobile phone terminal in which the link information is embedded is created and notified to the notification destination address received at the time of initialization. This enables a download request to each video clip from the mobile phone terminal that has received the notification information. Note that the video transmission unit 5009 has an HTTP server function and responds to an HTTP-based video clip download request from a mobile phone terminal.

  Next, the camera control unit 5006 interprets the PTZ sequence, creates a camera control command to be sent to the camera server 101, and transmits the created camera control command to the camera server 101 at the timing instructed by the PTZ sequence. Thus, camera control of the camera server 101 is performed. Then, when the interpretation of the PTZ sequence is completed, the video transmission unit 5009 is notified of the end of the PTZ sequence.

  With the above configuration, a user who uses a video clip viewer mounted on a mobile phone terminal can request a video clip from the video conversion server 400. Then, the function of the video conversion server 400 makes it possible for the video clip playback user to use the camera control state information at the time when the video clip is created.

  In particular, in the video clip viewer of the present embodiment, the camera control information part embedded as link information in the video clip is pre-read, the camera control content is analyzed, the camera control content is predicted on the video display screen, It is characterized in that it jumps to the forecasted video segment (video fast-forward) in response to an instruction from the user.

  In the present embodiment, an example in which the camera server is indirectly accessed via the video conversion server 400 is described. However, the viewer is based on the information (camera connection information and camera control information) obtained in step 602. The camera server 101 may be directly accessed.

  In the present embodiment, an example is described in which the video conversion server 400 is mounted on the network independently of the gateway 500 that connects the mobile phone network and the network. However, the video conversion server 400 is a part of the gateway 500. May be implemented as Also, it is possible to easily imagine a connection form in which the video conversion server and the gateway are connected by a dedicated line including a VPN (Virtual Private Network).

  In the present embodiment, an example in which the mobile phone viewer 200 communicates with the video conversion server 400 using HTTP communication is described. This communication is communication using SMTP (Simple Mail Transfer Protocol). There may be.

  Further, the HTTP communication and the SMTP communication between the mobile phone viewer 200 and the video conversion server 400 may use a secure communication path by using SSL (Secure Socket Layer) or the like together.

  In the present embodiment, an example in which the camera server 101 and the video conversion server 400 are independent has been described. However, the camera server 101 and the video conversion server 400 may be integrated. In that case, the video conversion server 400 may acquire the video data using the video capture board having the hardware configuration of the camera server 101 as in the video server of the camera server 101. In addition, even if the camera server 101 and the video conversion server 400 are an integrated video conversion server, it is easy to design so that video data can be acquired from video servers of other camera servers as in the present embodiment. I can imagine. As a result, there are effects such as load distribution of the encoding process of the camera server, load distribution of the distribution process, and prevention of congestion in the network communication infrastructure.

  In this embodiment, an example in which all camera control information in a video clip is prefetched has been described, but the amount of prefetching is not limited to the entire video clip. For example, a part of the video clip may be pre-read according to the memory capacity of the device that plays back the video clip (in this embodiment, a mobile phone terminal), and information detected within the range may be predicted.

  It can also be easily imagined that some camera control contents are selectively predicted from the camera control contents (types of camera control commands, see FIG. 9) and the accumulated values of fluctuations due to camera control. For example, in order to select the contents of camera control, an evaluation function using arguments of the accumulated variation value and the contents of camera control (camera control command type) may be introduced.

  For example, in the case of camera control with a large fluctuation on average such as preset control, a large value is returned, and in the case of camera control with a small fluctuation on average such as simple pan, tilt and zoom commands, a small value is returned. Use a function that returns. Further, it is possible to control such that a command having a variation value of a single command less than a certain threshold value is not displayed with respect to a threshold value obtained from a variation accumulation value obtained by accumulating variations due to all commands. FIG. 19 shows an example when the control of FIG. 9 is performed. In this case, the variation accumulated value in 100 seconds is 15, and control is performed so that the command [2] having a value lower than the average value (1.5) in time units is not displayed.

  In this embodiment, an example in which the video clip viewer operates on a mobile phone terminal has been described, but the operating environment is not limited to this. For example, unlike a PC or the like, use in an STB (Set Top Box) can be explained in the same manner in that it operates in an environment with many restrictions. In application to STB, a TV screen is used as a display screen, STB remote control input is used as a button for receiving input from a user, and CATV communication or Internet communication is used instead of mobile phone wireless communication as a communication medium. The point to use is different.

  In this embodiment, an example is described in which the video clip viewer pre-reads camera control information associated with each video segment of the video clip as link information when the video clip is played back. It can easily be imagined that the burden on the viewer during playback can be reduced by integrating the information into the header information of the video clip.

  Regarding the video segment division of FIG. 15B and the marking (video segment marker) for the video division point for direct display, there is a method based simply on the amount of data. An evaluation function can be used.

  That is, the time zone in which each control is concentrated and the average evaluation value is higher than the overall average is marked to be a short video clip (for example, applied in 3 and 4 in FIG. 15B), and the table value Can be marked to be a long video clip (for example, applied in 1 and 2 of FIG. 15B).

  In the present embodiment, an example is described in which video information and camera control information included in the same video clip are pre-read and used at the time of playback. However, this is not limited to the same video clip (file). It may be provided as a different file. Furthermore, video information and camera control information included in different video clips may be actively used in combination.

  In the present embodiment, an example in which the camera control content is predicted based on the relative time from the playback display time of the video clip being played back has been described, but the time to be predicted is not limited to the relative time from the display time. For example, it is easy to imagine using the relative time from the beginning of the video clip. In addition, when the video clip holds the generation time, it is conceivable to predict the camera control content at the absolute time.

(Second Embodiment)
In the second embodiment, as in the first embodiment, an example will be described in which a live video acquired from a camera server arranged on the Internet is converted into a video clip, transmitted, reproduced, and displayed.

  In particular, in this embodiment, a viewer for accumulated video clips that accumulates and browses a large number of video clip groups generated from live video of a large number of camera servers connected to the Internet has a large number of accumulated video clips. A feature is that camera control information is displayed in a list for a video clip of interest in the group, and a part thereof (a part detected as a breakpoint) is provided in a manner that is easy for the user to identify.

  Furthermore, in the present embodiment, the user selects a part of the camera control information list extracted from the video clip of interest and applies it to another video clip (drag and drop operation), thereby applying the applied video. It is characterized in that a live connection is made to the camera server that generated the clip, and the camera control content (camera control command) corresponding to the selected camera control information is requested.

  In the present embodiment, the network connection mode, hardware configuration, and the operation of each software are the same as described in the first embodiment. However, the hardware configuration for operating the viewer described with reference to FIG. 4 is not the mobile phone but the PC hardware (not shown). Further, the operation of the video clip viewer shown in FIG. 6 is different from that of the first embodiment.

  This embodiment is different from the first embodiment in that the video clip viewer operates in an environment with few restrictions such as a PC, and video clip selection and playback control are performed by an operation using a GUI. .

  The video clip viewer according to the second embodiment operates as a stored video clip viewer, and displays a created video clip stored in the operating hardware or the video storage server as a display target. The point that camera server connection information and camera control information are incorporated in these video clips is as described in the first embodiment.

  In the video clip playback / display processing of the stored video clip viewer, as shown in FIG. 17, a plurality of video clips can be played back simultaneously, and the camera incorporated as link information is the same as the video clip viewer of the first embodiment. Pre-read and forecast control information.

  Furthermore, for a video clip of interest (focused) among a plurality of video clips, a more detailed camera control content is displayed in a list.

  A part of the camera control information displayed in the list can be selected, and another video clip can be generated by drag-n-dropping to another video clip screen displayed in the multi-view of the stored video clip viewer. Live connection to the selected camera server, and the selected camera control information is applied to the camera server.

  FIG. 16 is a diagram illustrating the flow of the display operation of the video clip viewer according to the second embodiment.

  In step S1601, a plurality of video clips with camera control information held in the PC are acquired. These video clips are acquired to the video clip viewer in the PC through a route such as downloading from a network, attachment to a mail, or copying from a memory card. Typically, the created video clip stored in the video storage server is acquired.

  In step S1602, a video clip for prefetching and predicting camera control information is selected, and camera control information is prefetched and displayed in a list. Typically, one video clip is selected, but a plurality of video clips may be selected, and camera control information incorporated in these video clips may be used continuously. These are designated as video clip B1 to video clip Bn.

  In step S1603, a video clip (video clip A) for designating a camera server to be connected is selected. Typically, the video clip B1 to video clip Bn selected in step S1602 are dragged and dropped to the video clip A (FIG. 17).

  In step S1604, camera connection information incorporated in the video clip A selected above is extracted. In step S1605, a camera control PTZ sequence is extracted from the camera control information selected by the user in the video clip B1 to the video clip Bn.

  In step S1606, the obtained camera connection information is used for live connection to a designated camera server, and a PTZ sequence is requested as a camera control command.

  With the above configuration, a user using a video clip viewer mounted on a computer such as a PC can request a video clip from the video conversion server. The video clip playback user can use the camera control state information at the time when the video clip is created by the function of the video conversion server.

  In particular, in the present embodiment, in a video accumulation viewer that reproduces and displays a huge number of accumulated video clips, the camera control information incorporated in the video clips is pre-read and forecast information is displayed, so that these cameras are displayed. An example in which control information is partially selected and applied to a live video distribution connection to another camera server has been described.

  In the present embodiment, the camera control information extracted from the video clip is described as an example of controlling the camera server that is delivering live video, but this is not limited to live connection. For example, it can be easily imagined that a video clip is created again using the PTZ sequence of the extracted camera control information.

  In the present embodiment, an example is described in which automatic control is performed using a PTZ sequence (selected portion) of the extracted camera control information even when used for camera control of a camera server during live video distribution. This is not limited to automatic control. For example, in addition to automatic control, it is conceivable that the user interrupts automatic control and operates a PTZ sequence for camera control, such as fast-forward 1801, reverse-feed 1802, stop 1803, pause 1804, and playback 1805. Alternatively, the automatic control may be temporarily stopped 1084 and the PTZ sequence may be manually controlled using the scroll bar 1806 according to the user's preference (FIG. 18). In this case, the live control by the user operation and the automatic control by the PTZ sequence are mutually exclusive controlled.

(Other embodiments)
An object of the present invention is to supply a storage medium storing software program codes for realizing the functions of the above-described embodiments to a system or apparatus, and a computer (or CPU, MPU, etc.) of the system or apparatus stores the storage medium. It is also achieved by reading out and executing the program code stored in.

  In this case, the program code itself read from the storage medium realizes the novel function of the present invention, and the storage medium and program storing the program code constitute the present invention.

  The storage medium for supplying the program code is, for example, a flexible disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW. DVD-R, magnetic tape, nonvolatile memory card, ROM, and the like can be used.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an OS (operating system) running on the computer based on an instruction of the program code, etc. Includes a case where the function of each embodiment described above is realized by performing part or all of the actual processing.

  Further, after the program code read from the storage medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion is performed based on the instruction of the program code. This includes the case where the CPU or the like provided in the board or function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

It is the figure which showed the typical usage pattern using this invention. It is the figure which showed an example of the hardware constitutions of a video conversion server. It is the figure which showed an example of the hardware constitutions of a camera server. It is the figure which showed an example of the hardware constitutions which operate a viewer. It is the figure which modeled the part which the program comprises. It is a figure which shows the flow of operation | movement of the video clip viewer of a mobile telephone. It is the flowchart which showed operation | movement of the camera control server in a camera server. It is the flowchart which showed operation | movement of the video server in a camera server. It is the figure which illustrated the video clip reproduction display on a mobile telephone. It is the figure which illustrated the setting screen of the camera server. It is a flowchart which shows operation | movement of the setting program of a camera server. It is the figure which modeled the flow of the video data in a video conversion server. It is the flowchart which showed operation | movement of the video conversion server. It is a figure which shows the flow of a live video viewer. It is the figure which illustrated the video clip selection screen on a mobile telephone terminal. It is a figure which shows the flow of operation | movement of the video clip viewer of 2nd Embodiment. It is the figure which illustrated the drag and drop screen of 2nd Embodiment. It is the figure which illustrated the screen at the time of automatic control interruption of a 2nd embodiment. It is a figure for demonstrating the command selection by an evaluation function.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Camera server 102 Camera server 200 Viewer 300 Relay server 400 Video conversion server 500 Gateway 601 Mobile phone terminal 602 Mobile phone terminal

Claims (16)

A video display device for displaying a video file received from a camera server connected via a network,
Prior to video display of the video file, information extracting means for extracting camera control information contained in the video file;
Display means for displaying the video of the video file, and displaying camera control contents related to video in a time zone not displayed based on the camera control information extracted by the information extraction means;
A video display device comprising:   2. The video display device according to claim 1, wherein the extraction unit extracts connection information to a camera server and a camera control command as camera control information.   The video display apparatus according to claim 1, wherein the extraction unit extracts camera control information related to a video of all time included in the video file.   The video display apparatus according to claim 1, wherein the extraction unit extracts camera control information relating to a video of a part of time included in the video file.   3. The video according to claim 1, wherein the extraction unit extracts camera control information related to a video of a part of time included in the video file based on an available storage capacity of the video display device. Display device.   6. The video display device according to claim 5, wherein the display means applies an evaluation function to the extracted camera control information and displays the control content only when the evaluation value exceeds a set threshold value.   The display means applies the evaluation function together with the extracted camera control information and the history information including the camera fluctuation accumulated value, and displays the control content only when the evaluation value exceeds a set threshold value. The video display device according to claim 6.   2. The video display device according to claim 1, further comprising generating means for generating a video segment marker for directly displaying a video portion associated with camera control content.   Issue means for issuing a camera control command to the camera server, wherein the extraction means extracts connection destination camera server information from the video file, and the issue means issues a camera control command to the connection destination camera server. The video reproduction display device according to claim 1.   The video display apparatus according to claim 9, wherein the extraction unit extracts camera control information from a plurality of video files and uses the obtained camera control information in combination with each video file.   11. The video display apparatus according to claim 10, wherein when the extraction unit extracts information of one video file, the display unit displays a different video file.   12. The issuing means issues a selective automatic control command for selectively applying a part of the extracted camera control information to a camera server for automatic control. The video display device described.   13. The issuing unit according to claim 12, wherein the issuing unit issues a command for interrupting control by the camera control command, temporarily stopping the control, or performing a camera control operation based on a user input. Video display device. A control method of a video display device for displaying a video file received from a camera server connected via a network,
Prior to video display of the video file, an information extraction step of extracting camera control information included in the video file;
Displaying the video of the video file, and displaying the camera control content related to the video of the time zone not displayed based on the camera control information extracted by the information extraction step;
A method for controlling a video display device, comprising: A program for executing a control method of a video display device that displays a video file received from a camera server connected via a network,
Prior to video display of the video file, an information extraction step of extracting camera control information included in the video file;
Displaying the video of the video file, and displaying the camera control content related to the video of the time zone not displayed based on the camera control information extracted by the information extraction step;
A program comprising: A storage medium storing a program code for executing a control method of a video display device that displays a video file received from a camera server connected via a network,
Prior to video display of the video file, an information extraction code for extracting camera control information contained in the video file;
A display code for displaying the video of the video file, and displaying the camera control content related to the video of the time zone not displayed based on the camera control information extracted by the information extraction code;
A storage medium comprising:

Images