JP6627459B2 - Information transmission device, information processing system, transmission method and program - Google Patents

Description

  The present invention relates to an information transmission device, an information processing system, a transmission method, and a program.

  Cameras used in videoconferencing systems, monitoring systems, and the like usually have a pan / tilt / zoom function in order to photograph the entire imaging area. For pan / tilt, a mechanism for changing the imaging direction of a camera lens is generally mounted. In recent camera apparatuses, with an increase in resolution of a sensor and an advance in digital signal processing (DSP) technology, a pan-tilt function has been realized by digital processing of an image instead of a pan-tilt function by mechanical means. Was. Hereinafter, the pan / tilt function by digital processing is referred to as DPTZ (Digital Pan Tilt Zoom). DPTZ realizes an apparent pan-tilt function by digitally processing video data using a wide-angle (fish-eye) lens.

  In the DPTZ, since the imaging direction of the camera can be electrically switched, it is not necessary to change the imaging direction of the camera with a mechanical rotation mechanism. For this reason, if an imaging target is found, the imaging direction of the camera can be instantaneously changed. DPTZ is applied to, for example, an interlocking operation with a speaker tracking technology.

  Patent Document 1 proposes a digital monitoring system that improves operability of camera control without being affected by transmission delay via a network.

  However, in the related art such as Patent Literature 1, there is a case where the imaging direction of the camera cannot be appropriately controlled. For example, when a remote camera is operated via a network, a delay may occur from the time the operation is performed until the image after the operation is actually reflected on the camera is displayed. The delay includes a time delay required for the video captured by the camera to be transferred to the receiving device, and a time when a stop command is transmitted to the camera when a target is detected. Includes a time delay before the command is executed. Due to such a delay, for example, even if the panning operation is stopped when a target reaches the center of the screen while the camera is being panned, the target sent from the image that is sent thereafter has a higher position than the center. May be reflected too far.

  The present invention has been made in view of the above, and an object of the present invention is to provide an information transmission device, an information processing system, a transmission method, and a program that can more appropriately control the imaging direction of a camera.

In order to solve the above-described problem and achieve the object, the present invention provides a storage unit that stores an image captured by an imaging unit, and a receiving unit that receives a request to stop changing the imaging direction of an image from an external device. And transmitting, when the request is received, a specifying unit that specifies an image to stop changing the imaging direction among the images stored in the storage unit, and transmitting the image captured in the specified imaging direction. When the request is received, a transmitting unit that transmits an image captured in the imaging direction of the image specified by the specifying unit, and when the request is received, is captured after the specified image, A cutout section for cutting out a partial image corresponding to the specified imaging direction from the image stored in the storage section, and the transmitting section further transmits the partial image.

  ADVANTAGE OF THE INVENTION According to this invention, there exists an effect that it becomes possible to control the imaging direction of a camera more appropriately.

FIG. 1 is a block diagram illustrating a hardware configuration example of the video conference system according to the present embodiment. FIG. 2 is a diagram for explaining the operation principle of DPTZ. FIG. 3 is a diagram for explaining a problem of the remote pan-tilt operation. FIG. 4 is a block diagram illustrating an example of a functional configuration of the information processing apparatus according to the present embodiment. FIG. 5 is a flowchart illustrating an example of the video transmission process according to the present embodiment. FIG. 6 is a diagram showing a transition of frames of transmission video data. FIG. 7 is a diagram showing a transition of frames of transmission video data. FIG. 8 is a flowchart illustrating an example of the video receiving process according to the present embodiment. FIG. 9 is a diagram showing a transition of frames of received video data. FIG. 10 is a diagram showing a transition of frames of received video data.

  An embodiment of an information transmitting device, an information processing system, a transmitting method, and a program according to the present invention will be described below in detail with reference to the accompanying drawings.

  An outline of the information processing system of the present embodiment will be described. Hereinafter, a case where an information processing system is realized as a video conference system will be described as an example. The applicable system is not limited to the video conference system, but may be applied to a surveillance system using a camera.

  The video conference system includes a transmitting-side information processing device (information transmitting device) and a receiving-side information processing device (information receiving device). The information transmitting apparatus holds transmitted video data corresponding to the network delay time for recovery and updates the video data as needed. The resolution of the video data held for recovery is the entire area of the image sensor of the camera. In the case of DPTZ, the information transmitting device cuts out video data (partial image) corresponding to a specific area of the image sensor and transmits the video data to the information receiving device.

  When a command (stop command) for stopping pan / tilt is received from the information receiving device during the pan / tilt operation, the information transmitting device calculates the time (or frame) at which the stop command was issued by the information receiving device. That is, the information transmitting apparatus determines which frame video is being displayed by the information receiving apparatus when the pan / tilt stop operation is performed. Then, the information transmitting apparatus retransmits the video data after the time when the stop is instructed. The video data to be retransmitted may be video data stored for recovery. The information transmitting apparatus retransmits the subsequent video data in the pan / tilt direction (imaging direction) at the start of retransmission.

  As described above, the accuracy of the pan / tilt braking position can be improved by back-calculating the actual pan / tilt position when the stop command is issued by the information receiving device. Further, by providing a mechanism for distributing the recovery video from the time when the stop command is issued, the continuity of the video data can be maintained.

  When issuing the pan / tilt stop command, the information receiving device stops displaying the received video data at that time, and restarts the video display from the time when the reception of the recovery video is confirmed. When recording video data, a recovery video is placed immediately after the frame at the time of issuing the stop command. Thereby, even if the transfer of the video data is temporarily interrupted, for example, due to the blur adjustment of the pan / tilt position, the continuity of the output video data (for example, the reproduced video when recorded) is maintained. be able to.

  The video data is encoded and transmitted. In the video conference system, since real-time properties are prioritized, most frames are encoded as P pictures. In the case of a recovery video, an encoding process is performed in addition to the B picture. As a result, the recovery time can be reduced (that is, the code amount can be reduced).

  Hereinafter, an example in which the pan / tilt function is executed by the DPTZ will be described. However, a similar method can be applied to a case where the pan / tilt function is executed by mechanical means.

  FIG. 1 is a block diagram illustrating a hardware configuration example of the video conference system according to the present embodiment. As shown in FIG. 1, the video conference system includes information processing apparatuses 200A and 200B, a server 300, and a network 400.

  The information processing apparatuses 200 </ b> A and 200 </ b> B are arranged at different bases, respectively, and transmit and receive images and sounds via the server 300. Since the information processing apparatuses 200A and 200B have the same configuration, the information processing apparatuses 200A and 200B are simply referred to as the information processing apparatus 200 when there is no need to distinguish them. The information processing device 200 can be realized by any device such as a dedicated terminal for a video conference, a notebook PC (Personal Computer), a smartphone, and a tablet terminal.

  Hereinafter, the information processing apparatus 200A will be described as a transmitting terminal (information transmitting apparatus), and the information processing apparatus 200B will be described as a receiving terminal (information receiving apparatus).

  The server 300 relays transmission and reception of images (videos) and sounds between the plurality of information processing apparatuses 200. Further, the server 300 has a conference control function of controlling start and end of a conference between a plurality of information processing apparatuses.

  The network 400 is a network that connects the information processing device 200 and the server 300. The network 400 can be in any network form, such as the Internet, an intranet, and a LAN (Local Area Network).

  The information processing apparatus 200 includes a camera 10, a DPTZ control circuit 20, a video CODEC circuit 30, a video output processing circuit 40, a microphone array 50, an audio output circuit 60, an audio processing circuit 70, and a network (NW). It includes a processing circuit 80, an overall control circuit 90, an operation device 100, a RAM (Random Access Memory) 110, a recording device 120, a display device 130, and a speaker 140.

  The camera 10 is an imaging unit that captures an image of a participant in a video conference or the like. The camera 10 includes a lens 11, an image sensor 12, and an image processing circuit 13. The lens 11 is, for example, a wide-angle (fisheye) lens. The image sensor 12 converts an image collected through the lens 11 into an electric signal. The image sensor 12 can be realized by an image sensor such as a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD). The image processing circuit 13 performs signal processing such as shading and Bayer conversion on image data output from the image sensor 12.

  The camera 10 controls its own imaging range (imaging direction) by DPTZ. In order to enhance the effect of DPTZ, it is desirable that the lens 11 be an ultra-wide-angle lens and the image sensor 12 be equipped with a high-resolution sensor.

  A configuration may be adopted in which a plurality of cameras 10 are provided, and a video (panoramic shooting) obtained by blending the video captured by each camera 10 is transmitted to the DPTZ control circuit 20.

  The DPTZ control circuit 20 controls DPTZ. The DPTZ control circuit 20 includes a cutout circuit 21, an AFFINE (affine) conversion circuit 22, and a scaler 23. The operation principle of DPTZ by the DPTZ control circuit 20 will be described later.

  The video CODEC circuit 30 has a In accordance with standards such as H.264 / 265, encoding and decoding of video data are performed. Applicable standards are The present invention is not limited to H.264 / 265, and any conventionally used CODEC function can be applied.

  When transmitting video data (transmitted video data) to the information processing device 200 (a partner station) at another base, the video CODEC circuit 30 performs an encoding process on the video data input from the DPTZ control circuit 20. , And transfers the coded data to the NW processing circuit 80. When video data (received video data) is received from the partner station, the video CODEC circuit 30 takes in the encoded data transferred from the partner station via the NW processing circuit 80, decodes the encoded data, It is transferred to the video output processing circuit 40.

  The video output processing circuit 40 outputs the received video data decoded by the video CODEC circuit 30 or the video data captured by the own station to the display device 130. For example, when displaying video data of a plurality of sites, the video output processing circuit 40 performs a rendering process of a video to be displayed on the display device 130 according to layout setting information sent from the operation device 100 or the like.

  The microphone array 50 includes a microphone 51 and an A / D (analog / digital) converter 52. The microphone 51 collects sounds and the like uttered by the participants of the conference. The microphone array 50 collects sound in a plurality of microphones 51 in an array format. As a result, a function for specifying the direction of the speaker and a function for following the speaker can be mounted. When these functions are not provided, it is not necessary to provide the microphone 51 of the array type.

  The A / D converter 52 converts an analog signal of the sound collected by the microphone 51 into digital data and outputs the digital data to the sound processing circuit 70.

  The audio output circuit 60 converts audio data sent from another site into an analog signal and outputs the analog signal to the speaker 140. The audio output circuit 60 includes a D / A converter 61 that converts digital audio data into an analog signal.

  The audio processing circuit 70 performs various processes on the audio data. The audio processing circuit 70 includes a DSP (Digital Signal Processor) 71, an audio CODEC circuit 72, an NR (Noise Reduction) / EC (Echo Cancellation) circuit 73, an audio discrimination circuit 74, a speaker following detection circuit 75, It has.

  The DSP 71 performs digital signal processing on audio data. The DSP 71 takes in, for example, audio data from another site input via the NW processing circuit 80. The audio CODEC circuit 72 performs encoding and decoding of audio data. The NR / EC circuit 73 performs noise reduction processing and echo cancellation processing on audio data.

  In the case of audio data from another site, the DSP 71 transfers the processed audio data to the audio output circuit 60. At the same time, the DSP 71 recognizes the audio data transferred to the audio output circuit 60 and performs echo cancellation (EC) on the audio data wrapped around the microphone array 50 and input using the NR / EC circuit 73. Perform processing.

  The sound collected by the microphone array 50 is also input to the sound processing circuit 70. The audio determination circuit 74 extracts audio audio data from the audio data of the sound collected by the microphone 51. The speaker tracking detection circuit 75 executes a speaker tracking function using the extracted voice data or the like. The speaker tracking detection circuit 75 implements a speaker tracking function by applying a method of estimating the direction of a sound source from the voice collected by the microphone array 50, or the like. The DSP 71 transfers the processed audio data to the NW processing circuit 80 in synchronization with the video data.

  The NW processing circuit 80 transmits and receives video data and audio data. For example, the NW processing circuit 80 transmits the coded video data transferred from the video CODEC circuit 30 (the encoder 31) to the information processing apparatus 200 at another site via the network 400. Further, the NW processing circuit 80 captures the encoded data of the video transferred from the partner station via the network 400 and transfers the encoded data to the video CODEC circuit 30 (decoder 32).

  The NW processing circuit 80 may have a function of monitoring the band of the network 400 to determine an encoding parameter (such as a QP value) (a state detection circuit 81). Further, the NW processing circuit 80 may have a function of inputting information on the function and performance of the partner station in order to optimize the setting of the encoding parameter (such as the QP value) and the transmission mode (function discriminating circuit 82). .

  The overall control circuit 90 manages system control of the entire information processing apparatus 200. The overall control circuit 90 includes a CPU 91, a ROM (Read Only Memory) 92, a RAM 93, and an HDD 94.

  The HDD 94 stores various programs and data. For example, the HDD 94 stores an information processing program that controls the entire information processing apparatus 200. The HDD 94 manages programs and data using a predetermined file system and / or a DB (Data Base).

  The RAM 93 is a volatile semiconductor memory (storage device) that temporarily stores programs and data. The RAM 93 may be configured to be able to be shared with each unit other than the overall control circuit 90.

  The ROM 92 is a nonvolatile semiconductor memory (storage device) that can retain internal data even when the power is turned off. The ROM 92 stores, for example, data such as system settings of the information processing apparatus 200 and network-related settings.

  The CPU 91 realizes the overall control of the information processing apparatus 200 and the operation of various mounted functions, for example, by executing a program read on the RAM 93.

The functions of the overall control circuit 90 include the following functions.
-Mode setting and status management of each function according to instructions from the operator (participants of the conference, etc.)-Right to use RAM 93 and arbitration function for system (common) bus access right

  As for the pan / tilt operation, for example, the overall control circuit 90 has a function of notifying the DPTZ control circuit 20 of coordinate information of a cutout area accompanying the pan / tilt operation. The pan / tilt operation instruction is not limited to only a command from the operation device 100 by the operator of the own station. There may be a command related to a pan / tilt operation transmitted from the partner station (destination) via the network.

  The operation device 100 inputs various operations, and is realized by a keyboard, a mouse, a touch pad, a touch panel, and the like. For example, the operator instructs a pan / tilt operation using the operation device 100 while watching the video displayed on the display device 130. The pan / tilt operation instruction includes, for example, start of pan / tilt operation, stop of pan / tilt operation, pan / tilt direction, and the like.

  The instruction of the pan / tilt operation is sent to the overall control circuit 90. The overall control circuit 90 issues optimum settings and commands to each module while monitoring the status of other modules involved in the pan / tilt operation. For example, the overall control circuit 90 changes the start command of the pan / tilt operation, the stop command of the pan / tilt operation, and the direction of the pan / tilt operation in response to the start of the pan / tilt operation, the stop of the pan / tilt operation, and the instruction of the pan / tilt direction received from the operation device 100. Issue a direction instruction command to instruct.

  The operation device 100 is used for various settings and operation operations such as calling a participant to a conference.

  The RAM 110 is a storage medium that operates in conjunction with the DPTZ control circuit 20. The DPTZ control circuit 20 does not perform the cropping process (Cropping) on the video data sent from the camera 10 and accumulates the data of all areas in the RAM 110 every frame. When the storage area of the RAM 110 is insufficient, the oldest frames are deleted and new frames are sequentially accumulated. Thus, the RAM 110 can be configured to function as, for example, a ring buffer.

  Assuming that the capacity of the RAM 110 is the number of frames (= a) corresponding to the network delay time during video transmission and the number of frames (= b) corresponding to the network delay time during command transmission, at least (a + b) frames You need minutes.

  The recording device 120 records audio data and video data. For example, the recording device 120 records audio data output from the audio processing circuit 70 and video data output from the video output processing circuit 40 on a recording medium. As the recording medium, any medium such as a video tape, a DVD (Digital Versatile Disk), and a hard disk can be used. The recording device 120 can reproduce the conference scene by outputting the recorded data to the audio processing circuit 70 and the video output processing circuit 40.

  The display device 130 is a display unit that displays information such as video data. The display device 130 can be realized by any conventionally used display medium such as a liquid crystal display. The speaker 140 functions as an audio output unit that outputs audio data.

  The hardware configuration shown in FIG. 1 is an example, and the present invention is not limited to this. For example, part or all of each circuit may be realized by a dedicated hardware circuit such as an ASIC (Application Specific Integrated Circuit), or a program may be executed by a processing device such as a CPU (Central Processing Unit). That is, it may be realized by software. Alternatively, a circuit having some or all of the functions of a plurality of circuits may be used.

  Next, the operation principle of DPTZ by the DPTZ control circuit 20 will be described. FIG. 2 is a diagram for explaining the operation principle of DPTZ.

  Video data 501 captured by the camera 10 is input to the DPTZ control circuit 20. The video data 501 is wide-angle, large distortion, and high resolution video data. The video data 501 is simultaneously stored in the RAM 110 and may be reused for retransmission.

  The extraction circuit 21 of the DPTZ control circuit 20 extracts a designated area from the input video data 501. The area is specified using, for example, the operation device 100 of the own station or the partner station. The image (partial image 502) of the cut-out area (cut-out area) becomes a video displayed at another site. By dynamically changing the cutout area, an apparent pan / tilt operation is performed.

  The AFFINE conversion circuit 22 of the DPTZ control circuit 20 performs a distortion correction process on the partial image 502. As described above, since the image is captured by the ultra-wide-angle lens, the partial image 502 is an optically distorted image. Therefore, the AFFINE conversion circuit 22 performs distortion correction processing by AFFINE conversion according to the lens characteristics so that a natural image can be seen. The cutting process dynamically fluctuates in an arbitrary area. For this reason, it is necessary to replace the correction parameters of the AFFINE conversion in real time in accordance with the coordinates of the changing cutout area.

  The video data 503 from which the distortion has been removed by the correction processing by the AFFINE conversion circuit 22 is transferred to the video CODEC circuit 30. The video data 503 is subjected to encoding processing by the video CODEC circuit 30 and transmitted to the partner station via the network 400 in a stream data format.

  Next, problems in a pan-tilt operation (remote pan-tilt operation) in response to a request from a remote will be described. FIG. 3 is a diagram for explaining a problem of the remote pan-tilt operation.

  For example, the description will be made on the assumption that the information receiving apparatus stops pan-tilt operation in the N-th frame (N is an integer of 2 or more) of the transmission video data. Hereinafter, the i-th frame is referred to as frame i. Assuming that the number of frames corresponding to the network transfer delay time in video transmission (including the delay due to encoding and decoding, etc.) is a, the information transmitting apparatus already has a frame N at the time when the frame N is displayed on the information receiving apparatus. (N + a) is being transmitted. Furthermore, assuming that the number of frames corresponding to the network delay time in the transmission of the stop command is b, the information transmitting apparatus has already transmitted the frame (N + a + b) at the time when the stop command reaches the information receiving apparatus.

  As described above, since a delay corresponding to (a + b) frames occurs, frames (N + 1) to (N + a + b) are images in an area overrun from the target stop position. From the frame (N + a + b + 1), transmission in the same area as the frame N (= target stop position) becomes possible. However, for the information receiving apparatus, the frames (N + 1) to (N + a + b) are blank periods in which the video at the target stop position is not displayed.

  The present embodiment provides a mechanism capable of retransmitting the video data of the frame (N + 1) to the frame (N + a + b) at the target stop position. Since a blank period can be eliminated, for example, a temporary change in the shooting site can be prevented from being overlooked.

  FIG. 4 is a block diagram illustrating an example of a functional configuration of the information processing device 200 according to the present embodiment. As illustrated in FIG. 4, the information processing device 200 includes a storage unit 211, a receiving unit 201, a specifying unit 202, a cutout unit 203, an encoding unit 204, a decoding unit 205, a transmitting unit 206, A display control unit 207 and a control unit 208 are provided.

  The storage unit 211 stores video data (images) captured by the camera 10 (imaging unit). The storage unit 211 is realized, for example, by the RAM 110 in FIG.

  The receiving unit 201 receives various information transmitted from an external device such as the information processing device 200 as a partner station. For example, the receiving unit 201 receives a request (stop command) for stopping the change of the imaging direction of the video from the partner station.

  When the stop command is received, the specifying unit 202 specifies, out of the plurality of video data stored in the storage unit 211, the video data for which the change of the pan / tilt direction (imaging direction) is stopped. That is, the specifying unit 202 specifies, on the information receiving apparatus side, a frame specified as a frame for stopping the pan / tilt operation.

  The cutout unit 203 cuts out video data (partial image) corresponding to the specified pan-tilt direction from the video data captured by the camera 10. The extracting unit 203 is realized by, for example, the extracting circuit 21 in FIG.

  The encoding unit 204 encodes the clipped video data (partial image) for transmission to the partner station. The encoding unit 204 is realized by, for example, the encoder 31 in FIG. The decoding unit 205 decodes an image received from the partner station. The decoding unit 205 is realized by, for example, the decoder 32 in FIG.

  The transmission unit 206 transmits video data captured and encoded in the designated pan-tilt direction to the information processing apparatus 200, which is the partner station, via the NW processing circuit 80 or the like. Further, when the stop command is received, the transmission unit 206 transmits the coded video data captured and encoded in the pan-tilt direction corresponding to the image specified by the specifying unit 202 thereafter, to the information processing apparatus 200 that is the partner station. Send to The transmission unit 206 is realized by, for example, the DPTZ control circuit 20 of FIG.

  The display control unit 207 controls a process of displaying information such as video data on the display device 130. The display control unit 207 is realized by, for example, the video output processing circuit 40 in FIG.

  The control unit 208 controls the operation of the entire information processing device 200. The receiving unit 201, the specifying unit 202, and the control unit 208 are realized by, for example, the overall control circuit 90 in FIG.

  Next, a video transmission process performed by the information processing apparatus 200 according to the present embodiment thus configured will be described with reference to FIGS. FIG. 5 is a flowchart illustrating an example of the video transmission process according to the present embodiment. FIG. 5 shows a procedure on the information transmitting apparatus side remotely operated by the information receiving apparatus. 6 and 7 are diagrams showing the transition of frames of the transmission video data.

  When the information processing apparatus 200 is started, the control unit 208 performs an initial setting of each module, and sets a state where photographing by the camera 10 is possible (step S101). The control unit 208 sets an imaging mode of the camera 10 such as photometric conditions according to the environment of the conference (step S102). Thereafter, the control unit 208 sets the shooting mode of the camera 10 according to the manual setting of the operator.

  When the preparation is completed in steps S101 and S102, the control unit 208 requests the other station to start communication and starts a conference (step S103). The control unit 208 may start a conference in response to a communication request from the partner station. When recording a conference, the control unit 208 may activate the recording device 120 at this time.

  The receiving unit 201 determines whether a request to start a pan / tilt operation (remote pan / tilt request) has been received from the partner station (step S104). If not received (step S104: No), the process is repeated until it is received. During this time, imaging by the camera 10 is continued.

  When the remote pan / tilt request is received (step S104: Yes), the DPTZ control circuit 20 starts controlling the pan / tilt operation according to the pan / tilt direction instruction (direction instruction command) transmitted together with the remote pan / tilt request (step S105). For example, the extraction circuit 21 moves the extraction area according to the direction instruction command.

  The DPTZ control circuit 20 determines whether or not the cutout area has reached the end (limit position) of the displayable area (step S106). When the end is reached (Step S106: Yes), the DPTZ control circuit 20 stops the pan / tilt operation (Step S107), and thereafter transmits the video data of the cutout area at that time. In this case, the remote pan / tilt request is reset, and the process returns to step S104.

  If it is not the end (step S106: No), the receiving unit 201 determines whether a stop request (stop command) for the pan / tilt operation has been received from the partner station (step S108). If not received (step S108: No), the process returns to step S106 and the process is repeated. When the stop command is received (Step S108: Yes), the DPTZ control circuit 20 stops the pan / tilt operation (Step S109).

  The specifying unit 202 specifies a frame (hereinafter, referred to as a frame N) specified by the partner station as a frame for stopping the pan / tilt operation (step S110).

  The specifying unit 202 can specify the frame N by the following method. The specifying unit 202 grasps the network delay time of the video data and the command data in advance. For example, the identifying unit 202 transmits specific information such as dummy video data to the partner station at the start of the conference, and receives a reception time of receiving the specific information from the partner station. The specifying unit 202 can calculate the network delay time at the time of transmitting the video data from the transmission time and the reception time of transmitting the specific information. Similarly, by transmitting a dummy command as specific information, a network delay time at the time of command transmission can be calculated.

  The method for specifying the frame N is not limited to the above method, and the following method may be used. For example, the transmission unit 206 transmits transmission video data by adding identification information (a frame number or the like) for identifying a frame as metadata. The information receiving device acquires the frame number of the frame displayed when the stop command is issued, and transmits the acquired frame number to the information transmitting device together with the stop command. Thereby, the specifying unit 202 of the information transmitting device can specify the frame identified by the frame number as the frame N.

  Thereafter, the cutout unit 203 sets the cutout area to the same coordinates as the frame N. This is because when the frame N is displayed on the information receiving device side, it is determined that the receiving side has issued the stop command.

  The DPTZ control circuit 20 determines whether the frame transition mode is mode 1 (step S111). The frame transition mode indicates how to set a frame that transitions after the frame N. The frame transition mode includes, for example, mode 1 and mode 2. Mode 1 is a mode in which the image receiving apparatus returns to the frame at the time when the stop command is issued and retransmits the video data. Mode 2 is a mode in which the video data of the frame is transmitted in synchronization with the video data captured by the camera 10 without returning to the frame at the time when the stop command is issued by the information receiving device.

  The frame transition mode may be set, for example, according to an instruction from the operator, and may be stored in a storage unit such as the RAM 93. The DPTZ control circuit 20 makes the determination in step S111 with reference to the frame transition mode set as described above.

  When the frame transition mode is the mode 1 (step S111: Yes), the DPTZ control circuit 20 retransmits the video data using the video data stored in the RAM 110 as a target image (transmission video data) of the cutout processing (the transmission video data). Step S112). The frame to start retransmission is frame (N + 1).

  In the case of mode 1, it is necessary to store frames from the frame (N + 1) to the frame (N + a + b) at the time when the stop command is received in the RAM 110. In other words, the capacity of the RAM 110 needs at least (a + b) frames.

  Since the video data is transmitted from the camera 10 even while the video data is being retransmitted, the DPTZ control circuit 20 stores the video data from the camera 10 in the RAM 110. At the start of retransmission, the actual transmission video data has a delay of (a + b) frames with respect to the camera. Therefore, it is necessary to increase the transmission speed (transmission frame rate) of the transmission video data from the imaging speed of the camera 10 to recover the delay with respect to the camera 10.

  Therefore, the encoding unit 204 (encoder 31) sets the encoding efficiency of the image to be retransmitted higher than the encoding efficiency of the normal image (high compression mode). For example, the encoding process is usually performed only with P pictures during encoding, but during periods when the transmission frame rate needs to be improved, B pictures are also inserted to increase the encoding efficiency. If the coding efficiency is not sufficiently increased due to the network bandwidth, the frame delay with respect to the camera 10 may be recovered by adjusting the QP value or lowering the image resolution.

  The DPTZ control circuit 20 determines whether the transmission video data has caught up with the video data output from the camera 10 (Step S113). The DPTZ control circuit 20, for example, determines that it has caught up with the video data when all the video data stored in the ring buffer (RAM 110) has been processed.

  If not (step S113: No), the process returns to step S112 and the retransmission process is repeated. When catching up (Step S113: Yes), the DPTZ control circuit 20 performs processing so that the video data output from the camera 10 is used as the target image (transmission video data) of the cutout processing instead of the video data stored in the RAM 110. Is switched (step S114). Note that the encoding unit 204 cancels the setting of the high compression mode and executes the encoding process with normal encoding efficiency.

  FIG. 6 shows the frame transition of the transmission video data in the mode 1. In the example of FIG. 6, a pan / tilt stop command is received after the transmission of the frame (N + a + b). A period 601 indicates a period until the retransmitted video data catches up with the video data from the camera 10. In the period 601, transmission video data is generated from the video data stored in the RAM 110. The pan / tilt direction of the video data to be retransmitted is the same as the pan / tilt direction of frame N. Note that the frame with the symbol * indicates that it is a frame cut out from the video data stored in the RAM 110. The frame number when catching up with the video data from the camera 10 is (N + x). x is an integer satisfying x> a + b.

  Returning to FIG. 5, if it is determined in step S111 that the frame transition mode is not mode 1, that is, it is mode 2 (step S111: No), the DPTZ control circuit 20 cuts out the video data output from the camera 10. The transmission processing of the transmission video data is continued as an image to be processed (transmission video data) (step S115).

  FIG. 7 shows a frame transition of transmission video data in the case of mode 2. As in FIG. 6, an example is shown in which a pan / tilt stop command is received after transmission of the frame (N + a + b). In the case of mode 2, after the frame (N + a + b + 1), the frame of the video data in which the pan / tilt operation has been stopped is transmitted. The pan / tilt direction at this time is the same as that of the frame N.

  Returning to FIG. 5, after step S114 and step S115, the process returns to step S104 and the process is repeated. This returns to the normal state.

  Next, processing on the video receiving device side will be described. FIG. 8 is a flowchart illustrating an example of the video receiving process according to the present embodiment. FIGS. 9 and 10 are diagrams showing the transition of frames of the received video data.

  Steps S201 to S203 are the same as steps S101 to S103 in FIG. 5 which is a flowchart on the information transmitting apparatus side, and thus description thereof is omitted.

  The operator determines whether a pan-tilt operation is necessary, for example, while watching the image displayed on the display device 130. When a pan-tilt operation is necessary, the operator instructs the start of the pan-tilt operation using, for example, the operation device 100. The control unit 208 receives the start of the remote pan / tilt operation by the operation device 100 (Step S204). The control unit 208 issues a remote pan / tilt operation start command and a direction instruction command in response to the start of the remote pan / tilt operation and the instruction of the pan / tilt direction (step S205).

  The control unit 208 determines whether or not the video data at the end (limit position) of the displayable area on the information transmission device side has been received (step S206). When the video data at the end is received (step S206: Yes), the control unit 208 invalidates the pan / tilt operation (step S207). For example, the control unit 208 stops (initializes) transmission of a command related to the pan / tilt operation. Thereafter, the process returns to step S204 and the process is repeated. If the pan / tilt direction is changed, the remote pan / tilt operation becomes valid again.

  If the video data at the end has not been received (step S207: No), the control unit 208 determines whether or not an instruction to stop the pan / tilt operation by an operator or the like has been received (step S208). If the stop instruction has not been received (step S208: No), the process returns to step S106 and the process is repeated.

  When the stop instruction is received (Step S208: Yes), the control unit 208 issues a remote pan / tilt stop command (Step S209). When the frame number is added to the video data as metadata, the control unit 208 obtains the frame number of the frame displayed at the time of issuing the stop command, and sends the obtained frame number to the partner station together with the stop command. Send. Thereby, the information transmitting apparatus can specify the frame (frame N) at the time when the stop command is issued using the frame number.

  The control unit 208 determines whether the frame transition mode is mode 1 (step S210). When the frame transition mode is mode 1 (step S210: Yes), the control unit 208 temporarily sets the video data to be displayed to a still image (step S211). For example, the control unit 208 controls the video data of the frame N to be displayed in a still state. The control unit 208 may discard the frames (N + 1) to (N + a + b) received thereafter.

  The control unit 208 determines whether or not the frame (N + 1) has been received (step S212). For example, the control unit 208 determines whether or not the frame (N + 1) has been received using the frame number added as the metadata. For example, when the frame rate of the received video data is changed, the control unit 208 may determine that the frame (N + 1) has been received.

  If the frame (N + 1) has not been received (step S212: No), the process is repeated until the frame (N + 1) is received. When the frame (N + 1) is received (Step S212: Yes), the control unit 208 cancels the function of setting the video data to be displayed as a still image, and restarts the display of the video data from the frame (N + 1) (Step S213). ). After the restart, video data is transmitted from the information transmitting device at a high frame rate. Therefore, the control unit 208 increases the frame rate of display on the display device 130 so as to follow the control. The control unit 208 displays the frame after returning to the normal frame rate, that is, the frames after the frame (N + x + 1) at the normal frame rate. The information transmitting apparatus transmits video data at a normal frame rate after the frame (N + x + 1).

  FIG. 9 shows a frame transition of the received video data in the mode 2. FIG. 9 shows an example in which a stop command is issued at the same timing as in FIG. 6 showing the transition of the frame of the transmission video data. A period 901 indicates a period in which display of video data is stopped (for example, a still image is displayed). A period 902 indicates a period until the video data to be retransmitted catches up with the video data from the camera 10. In the period 902, the display frame rate is set high in accordance with the frame rate of the received video data. Data to be recorded on the recording device 120 does not need to be adjusted because real-time properties are not required. For example, the video data received in the period 901 may be recorded on the recording device 120 without being destroyed or before being destroyed.

  Returning to FIG. 8, when it is determined in step S210 that the frame transition mode is not mode 1, that is, mode 2 (step S210: No), the control unit 208 continues to receive and display normal video data. (Step S214).

  FIG. 10 shows a frame transition of the received video data in the mode 2. As in FIG. 7, an example is shown in which a pan / tilt stop command is received after the transmission of the frame (N + a + b). A period 1001 indicates a period during which video data at a position overrun from the designated stop position is displayed. As in the case of the mode 1, the display of the video data may be stopped in the period 1001. Since the video data (frame (N + 1) to frame (N + a + b)) received in the period 1001 corresponds to video data overrun from the requested stop position, the video data may not be stored in the recording device 120. After the frame (N + a + b + 1), the frame of the video data in which the pan / tilt operation has been stopped is transmitted. The pan / tilt direction at this time is the same as that of the frame N.

  Returning to FIG. 8, after step S213 and step S214, the process returns to step S204 and the processing is repeated. This returns to the normal state.

  The program executed by the apparatus according to the present embodiment is provided by being incorporated in a ROM or the like in advance.

  The program executed by the apparatus of the present embodiment can be read by a computer such as a CD-ROM, a flexible disk (FD), a CD-R, and a DVD (Digital Versatile Disk) in an installable or executable file. It may be configured to be recorded on a simple recording medium and provided as a computer program product.

  Further, the program executed by the apparatus of the present embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Further, the program executed by the apparatus of the present embodiment may be provided or distributed via a network such as the Internet.

  The program executed by the apparatus according to the present embodiment has a module configuration including the above-described units. As actual hardware, the CPU (processor) reads out the program from the ROM and executes the program. It is loaded on the storage device, and each unit is generated on the main storage device.

Reference Signs List 10 camera 11 lens 12 image sensor 13 image processing circuit 20 DPTZ control circuit 21 extraction circuit 22 AFFINE conversion circuit 23 scaler 30 video CODEC circuit 31 encoder 32 decoder 40 video output processing circuit 50 microphone array 51 microphone 52 A / D converter 60 audio Output circuit 61 D / A converter 70 Audio processing circuit 71 DSP
72 audio CODEC circuit 73 NR / EC circuit 74 audio discrimination circuit 75 speaker following detection circuit 80 NW processing circuit 90 overall control circuit 91 CPU
92 ROM
93 RAM
94 HDD
100 Operating device 110 RAM
120 Recording device 130 Display device 140 Speaker 200 Information processing device 201 Receiving unit 202 Identifying unit 203 Extracting unit 204 Encoding unit 205 Decoding unit 206 Transmission unit 207 Display control unit 208 Control unit 211 Storage unit 300 Server 400 Network

JP 2008-252331 A JP 2015-095803 A

Claims (7)

A storage unit that stores an image captured by the imaging unit,
A receiving unit that receives a request to stop changing the image capturing direction from an external device,
When the request is received, of the images stored in the storage unit, a specifying unit that specifies an image to stop changing the imaging direction,
A transmitting unit that transmits an image captured in the specified imaging direction and, when the request is received, transmits an image captured in the imaging direction of the image specified by the specifying unit.
When the request is received, the image is captured after the specified image, from the image stored in the storage unit, a cutout unit that cuts out a partial image corresponding to the specified imaging direction,
The transmitting unit further transmits the partial image,
Information transmission device. An encoding unit that encodes a partial image with a higher encoding efficiency for a partial image cut out from an image after the specified image than a partial image cut out from an image before the specified image. In addition,
The transmitting unit transmits the encoded partial image,
The information transmitting device according to claim 1 . The identification unit transmits identification information to the external device,
The receiving unit receives a reception time of receiving the specific information from the external device,
The specifying unit is configured to specify an image to stop changing the imaging direction based on a time from a transmission time at which the transmission unit transmits the specific information to the reception time,
The information transmitting device according to claim 1. The transmitting unit transmits an image with identification information of the image added thereto,
The receiving unit receives the request to which the identification information is added,
The specifying unit specifies an image identified by the identification information added to the request as an image to stop changing the imaging direction,
The information transmitting device according to claim 1. An information processing system including an information transmitting device and an information receiving device,
The information transmitting device,
A storage unit that stores an image captured by the imaging unit,
A first receiving unit that receives a request to stop changing the imaging direction of an image from the information receiving device;
When the request is received, of the images stored in the storage unit, a specifying unit that specifies an image to stop changing the imaging direction,
A first transmission unit that transmits an image captured in the specified imaging direction and, when the request is received, transmits an image captured in the imaging direction of the image specified by the specification unit ;
When the request is received, the image is captured after the specified image, from the image stored in the storage unit, a cutout unit that cuts out a partial image corresponding to the specified imaging direction,
The first transmission unit further transmits the partial image,
The information receiving device,
A second transmission unit that transmits the request to the information transmission device;
A second receiving unit that receives an image from the information transmitting device;
A display control unit that causes the display unit to display the received image,
Information processing system. A receiving step of receiving a request to stop changing the image capturing direction from an external device,
When the request is received, a specifying step of specifying an image to stop changing the imaging direction among images stored in the storage unit that stores the image captured by the imaging unit;
Transmits the images captured with the specified image capturing direction, when the request is received, a first transmission step of transmitting the image captured by the imaging direction of the specified image by the specifying step,
If the request has been received, the image is captured after the specified image, from the image stored in the storage unit, a cutout step of cutting out a partial image corresponding to the specified imaging direction,
A second transmitting step of transmitting the partial image;
A transmission method comprising: On the computer,
A receiving step of receiving a request to stop changing the image capturing direction from an external device,
When the request is received, a specifying step of specifying an image to stop changing the imaging direction among images stored in the storage unit that stores the image captured by the imaging unit;
Transmits the images captured with the specified image capturing direction, when the request is received, a first transmission step of transmitting the image captured by the imaging direction of the specified image by the specifying step,
If the request has been received, the image is captured after the specified image, from the image stored in the storage unit, a cutout step of cutting out a partial image corresponding to the specified imaging direction,
A second transmitting step of transmitting the partial image;
The program to execute.

Images