JP6360300B2 - COMMUNICATION DEVICE, IMAGING DEVICE, ITS CONTROL METHOD, AND PROGRAM - Google Patents

Description

  The present invention relates to live streaming between two devices via a network.

  Conventionally, by transferring video captured by a camera by live streaming to a user's terminal, the user can view video and audio that are sequentially sent from a remote camera to his / her terminal. In addition, there are a self-photographing mode for taking a picture of yourself and a remote-photographing mode for taking pictures of other people and animals.

  In the self-portrait mode, it is difficult to pick up the user's voice with the microphone of the remote camera. For example, as disclosed in Patent Document 1, the video captured by the camera is displayed on the remote controller of the camera and voice is input. A device with a microphone is required. Thereby, the user can record the audio picked up by the microphone in synchronization with the video data by speaking to the microphone of the remote controller while watching the video sent from the camera. In Patent Document 1, it is necessary to check the angle of view of the camera using the video displayed on the remote controller and match the sound with the actual video.

  In the remote shooting mode, if a delay occurs due to communication conditions, a time difference occurs between the actually shot video and the displayed video, so it is necessary to match the sound with the displayed video. . Here, there is an apparatus that calculates a voice arrival time based on a distance from a subject to a microphone and a sound speed, and corrects a lip sync shift caused by a distance between the video and the sound reaching the microphone (for example, Patent Document 2). ).

Japanese Patent Application Laid-Open No. 11-155092 JP 2010-258568 A

  By the way, in the said patent document 1, since the delay of communication is not considered, when a delay generate | occur | produces, an image | video and an audio | voice will stop synchronizing. Further, in Patent Document 2, the voice arrival time is calculated based on the distance from the subject to the microphone and the sound speed, and it is not possible to make settings according to the usage such as the self-shooting mode and the remote shooting mode.

  The present invention has been made in view of the above problems, and its purpose is to determine whether to synthesize and record the video obtained on the server side and the audio obtained on the client side according to communication delay and usage in live streaming. By doing so, the technology to optimize the synchronization of video and audio is realized.

  In order to solve the above-described problems and achieve the object, a communication device of the present invention is a communication device capable of communicating with an imaging device, receiving means for receiving video data from the imaging device in a predetermined cycle, and the reception Display means for displaying video data received by the means, input means for inputting audio data, setting means for setting a shooting mode, video data received by the receiving means, and audio data input by the input means According to the recording means for combining and recording, the shooting mode set by the setting means, and the delay until the video shot by the imaging device is displayed on the display means, the reception means has received And control means for controlling the recording means so that the video data and the audio data input by the input means are recorded synchronously.

  According to the present invention, in live streaming, synchronization between video and audio is determined by deciding whether to synthesize and record video obtained on the server side and audio obtained on the client side according to communication delay and usage. Can be optimized.

1 is a block diagram illustrating a configuration of an imaging apparatus according to an embodiment of the present invention. The block diagram which illustrates the composition of the communication apparatus of the embodiment concerning the present invention. The figure explaining the system and live streaming operation | movement of this embodiment. The flowchart which shows operation | movement of the communication apparatus of this embodiment. 6 is a flowchart showing the operation of the imaging apparatus according to the present embodiment. FIG. 6 illustrates a live streaming operation according to the second embodiment. The figure which illustrates the play list of Embodiment 2. The figure which illustrates the play list in communication of Embodiment 2. FIG. 9 is a flowchart showing the operation of the communication apparatus according to the second embodiment. The figure which illustrates the live streaming screen in the communication apparatus of this embodiment.

  EMBODIMENT OF THE INVENTION Below, the form for implementing this invention with reference to an accompanying drawing is demonstrated in detail. The embodiment described below is an example for realizing the present invention, and should be appropriately modified or changed depending on the configuration and various conditions of the apparatus to which the present invention is applied. It is not limited to the embodiment.

  Hereinafter, the present invention is based on a system in which an imaging device 1000 as a server and a communication device 2000 as a client are communicably connected so that video captured by the imaging device 1000 can be viewed by live streaming on the communication device 2000. An example will be described. In this embodiment, an example in which the imaging apparatus 1000 is applied to a digital video camera and the communication apparatus 2000 is applied to a communication terminal such as a smartphone will be described. However, the present invention is not limited to this. For example, the present invention is applied to an information processing device such as a personal computer, a mobile phone, a tablet device terminal, a television, and the like as a communication device, such as a camera-equipped mobile phone or a portable media player, a so-called tablet terminal. it can.

  <Arrangement of Imaging Apparatus> With reference to FIG. 1, the outline of the arrangement and functions of an imaging apparatus according to an embodiment of the present invention will be described.

  In the imaging apparatus (hereinafter referred to as camera) 1000 shown in FIG. 1, a CPU (Central Processing Unit) 1001, a ROM (Read Only Memory) 1002, a RAM (Random Access Memory) 1003, an input processing unit 1004, an output with respect to an internal bus 1010. A processing unit 1006, a communication control unit 1008, a recording medium control unit 1011, a camera signal processing unit 1015, and an encoding / decoding processing unit 1016 are connected. Each unit connected to the internal bus 1010 can exchange data with each other via the internal bus 1010.

  The ROM 1002 stores various programs and setting data for the CPU 1001 to operate. A flash memory is also included. The RAM 1003 appropriately stores programs and variables necessary for the operation of the CPU 1001, temporary work data, and the like.

  The CPU 1001 controls each part of the camera 1000 using the RAM 1003 as a work memory according to a program stored in the ROM 1002 or the recording medium 1012.

  The optical system 1013 is a photographing lens including a focus, a diaphragm mechanism, and the like, and forms an optical image of a subject. The imaging element 1014 is configured by a CCD, a CMOS element, or the like, and here includes an A / D converter, which converts an optical image into an analog electric signal and then converts it into a digital signal.

  Based on the control of the CPU 1001, the camera signal processing unit 1015 performs resize processing such as predetermined pixel interpolation / reduction, color conversion, various correction processing, and the like on the digital signal converted by the image sensor 1014.

  Based on the control of the CPU 1001, the encoding / decoding processing unit 1016 performs compression encoding on the digital signal processed by the camera signal processing unit 1015 at a predetermined bit rate and format, or decoding video compression encoded data.

  The audio is not specifically shown. However, when distributing video with audio, a microphone, an A / D converter for digitizing an analog analog signal, and a configuration for encoding digital data are provided. . Naturally, at the time of video recording, audio and video are simultaneously recorded, and the video / audio is multiplexed by the encoding / decoding processing unit 1016 to generate video data with audio.

  The input processing unit 1004 receives a user operation on the operation unit 1005, generates a control signal corresponding to the operation, and supplies the control signal to the CPU 1001. For example, the operation unit 1005 includes character information input devices such as a keyboard and pointing devices such as a mouse and a touch panel as input devices that receive user operations. In addition, remote control devices such as an infrared remote controller are also included. The touch panel is an input device that outputs coordinate information according to a position touched with respect to, for example, a planar input unit. Thereby, it is possible to cause the camera 1000 to perform an operation according to the user operation.

  The output processing unit 1006 outputs a display signal to be displayed on the display unit 1007 based on display data such as GUI (Graphical User Interface) generated by the CPU 1001 according to the program.

  Note that when a touch panel is used as the operation unit 1005, the operation unit 1005 and the display unit 1007 can be configured integrally. For example, the touch panel is configured so that light transmittance does not hinder display on the display portion 1007 and is attached to the upper layer of the display surface of the display portion 1007. Then, the input coordinates on the touch panel are associated with the display coordinates on the display unit 1007. Thereby, it is possible to configure a GUI as if the user can directly operate the screen displayed on the display unit 1007.

  The recording medium control unit 1011 is connected to a recording medium 1012 such as an HDD or a non-volatile semiconductor memory, and reads data from the connected recording medium 1012 and writes data to the recording medium 1012 based on the control of the CPU 1001. I do. The recording medium 1012 to which the recording medium control unit 1011 can be connected may be a detachable nonvolatile semiconductor memory such as a memory card via a socket (not shown).

  The recording medium 1012 can record information necessary for the control of the CPU 1001 in addition to the captured video data.

  The communication control unit 1008 communicates with the communication device 2000 and other external devices via a connector (wired) / antenna (wireless) 1009 based on the control of the CPU 1001. As a communication method, wireless IEEE802.11, Bluetooth (registered trademark), wired IEEE802.3, or the like can be used.

  The hardware configuration is not limited to that shown in FIG. 1. For example, one piece of hardware may perform display control, communication control, recording control, image processing control, and the like and function as each unit of the camera 1000. . A plurality of hardware may function as one means in cooperation.

  <Configuration of Communication Device> Next, the configuration and functions of the communication device according to the embodiment of the present invention will be outlined with reference to FIG.

  In the communication apparatus (hereinafter referred to as a terminal) 2000 shown in FIG. 2, many parts are the same as those of the camera 1000, and thus descriptions of the CPU 2001 to the recording medium 2012, which are the same as those of the camera 1000, are omitted. The encoding / decoding processing unit 2013 performs decoding of the video compression encoded data and re-encoding of the decoded data as necessary based on the control of the CPU 2001. The microphone 2014 converts sound vibrations into electrical signals as sound collection means. The audio signal processing unit 2015 converts the audio signal converted by the microphone 2014 into a digital signal, and performs various correction processes. Based on the control of the CPU 2001, the encoding / decoding processing unit 2013 encodes the audio signal generated by the audio signal processing unit 2015, and performs video and audio multiplexing processing.

  <Description of Live Streaming Operation> Next, a live streaming operation using a frame of a JPEG image by the system of the present embodiment will be described with reference to FIGS.

  3A, when the user executes the live streaming mode from the operation unit 1005 in the camera 1000, the camera 1000 sets the communication control unit 1008 in a communicable state under the control of the CPU 1001.

  Further, the user operates the operation unit 2005 on the terminal 2000 to activate an application necessary for communication connection processing and live streaming. In response to this, the CPU 2001 of the terminal 2000 controls the communication control unit 2008 according to the program stored in the ROM 2002 or the recording medium 2012, starts communication with the camera 1000, and performs connection processing.

  Here, the camera 1000 and the terminal 2000 use HTTP (Hypertext Transfer Protocol) / TCP (Transmission Control Protocol) as a communication protocol. Further, it is assumed that the communication connection supports UPnP (Universal Plug and Play). When the UPnP compatible terminal 2000 connects the device to the network, the IP (Internet Protocol) address is set by DHCP (Dynamic Host Configuration Protocol) or AutoIP. In order to recognize each other device on the network, the device that has acquired the IP address acquires information such as device search, response device type, and service function by “device discovery and control” (step 3002). . In response to the device search request from the terminal 2000, the camera 1000 responds with device information, frame acquisition destination information of device specific information, and the like (step 3003).

  When the connection process between the camera 1000 and the terminal 2000 is completed, the camera 1000 starts live streaming.

  In live streaming, the CPU 1001 of the camera 1000 starts signal output from the image sensor 1014, processes the output into an appropriate frame 3001 by the camera signal processing unit 1015, and sends the frame 3001 to the encoding / decoding processing unit 1016. .

  The encoding / decoding processing unit 1016 compresses and encodes the received frame 3001 at a predetermined bit rate and format, and stores it in the RAM 1003 or the recording medium 1012.

  The CPU 1001 of the camera 1000 updates the frame 3001 every predetermined period T. In the following description, it is assumed that T = 1/30 seconds.

  The CPU 1001 generates path information associated with the storage destination of the frame 3001. The path information is used as acquisition destination information when the terminal 2000 acquires a frame.

  It is assumed that the terminal 2000 stores a predetermined cycle T in advance or can obtain it at the time of device information acquisition by including it in the device information of the camera 1000.

  The terminal 2000 makes a frame acquisition request (HTTP GET method) to the frame acquisition destination acquired in step 3003 about T (seconds) after the start of live streaming (step 3004).

  The camera 1000 transmits the requested frame 3001 as a response frame (step 3005).

  The terminal 2000 passes the received frame 3001 to the encoding / decoding processing unit 2013, decodes it, and performs playback display on the display unit 2007 via the output processing unit 2006. When the terminal REC10007 is executed by the live streaming application, the decoded data or the data portion excluding the header from the frame 3001 is recorded on the recording medium 2012, and the sequentially received frame and the audio from the audio signal processing unit 2015 are recorded. Each signal is encoded, multiplexed, synthesized, and recorded.

During streaming, the camera 1000 performs frame update about every T (seconds). The acquired frame is deleted.
The terminal 2000 makes a frame acquisition request about every T (seconds) (step 3004).

  Note that a unique ID of the terminal or application is added to the request processing of the terminal 2000 (step 3004). Then, the camera 1000 of this embodiment performs streaming only for the request for the ID requested first. That is, it is assumed that the camera 1000 and the terminal 2000 of this embodiment perform streaming only with a one-to-one connection.

  According to the system of the present embodiment, in the self-portrait mode, the photographer synchronizes the video and audio by speaking to the microphone 2014 of the terminal 2000 while watching the video of himself / herself sent from the camera 1000. Can be recorded. Further, in the remote shooting mode, as shown in FIG. 3B, the photographer 3101 can input sound in accordance with the displayed video while viewing the video of the subject 3102 shot by the camera 1000. If the camera is equipped with a microphone, in the remote shooting mode, the video captured by the camera 1000 and the sound picked up by the microphone are sent to the terminal 2000. Audio can be input while viewing.

  FIG. 10 shows an example of an application screen in live streaming on the terminal 2000, in which an operation unit 2005 and a display unit 2007 are integrated. The live streaming data is displayed in a display area 10001, and the status of the camera 1000 displays zoom position information 10002, recording state 10003, battery information 10004, and the like. Further, as operation means, a zoom instruction 10005, a camera REC10006, a terminal REC10007, a selfie / remote shooting 10008, and the like are provided. By selecting self-shooting / remote shooting 10008, the user can set a shooting mode of either the self-shooting mode or the remote shooting mode before starting recording of video and audio.

  <Description of System Operation> Next, with reference to FIGS. 4 and 5, the operation of the camera 1000 and the terminal 2000 constituting the system of this embodiment will be described.

  First, a process executed by the CPU 2001 after the terminal 2000 establishes connection with the camera 1000 will be described with reference to FIG.

  In step S4001, the frame acquisition destination information is acquired from the device / frame acquisition destination information (step 3002) acquired at the time of connection and held.

  In step S4002, it is determined whether the terminal REC10007 has been executed by the live streaming application. Note that this determination is not limited to whether the terminal REC10007 has been executed, but can be determined to start recording when the camera 1000 detects the target subject. If it has been executed, the process advances to step S4003. If not, or if it has already been executed and recording has started, the process advances to step S4007.

  In step S4003, voice input from the microphone 2014 is started.

  In step S4004, it is determined whether the setting of self-shooting / remote shooting 10008 is the self-shooting mode or the remote shooting mode. In the case of the self-photographing mode, the process proceeds to step S4005. In the case of the remote-photographing mode, the process proceeds to step S4006. Note that the mode setting is set in advance by the user via the operation unit 2005. In addition, there is a terminal 2000 that does not set the mode from the operation unit 2005 and sets the self-portrait mode when the user's line of sight is detected within the angle of view of the camera 1000 or communicates within the angle of view of the camera 1000 When doing so, it is possible to set the self-portrait mode.

  In step S4005, a request for adding start point information to the frame at the start of recording is transmitted. In the self-portrait mode, it is necessary to synchronize audio data with the frame processed by the camera signal processing unit 1015 of the camera 1000.

  In step S4006, the frame displayed on the display unit 2007 and the audio data are combined and recording on the recording medium 2012 is started. In the case of the remote shooting mode, it is necessary to synchronize the audio data with the frame displayed on the display unit 2007.

  In step S4007, it is determined whether a predetermined time has elapsed. If the predetermined time has elapsed, the process proceeds to step S4008, and if not, step S4002 is repeated. Here, the predetermined time is desirably set to a value equivalent to the predetermined period T generated by the camera 1000.

  In step S4008, a frame acquisition request (step 3004) is made using the frame acquisition destination information acquired in step S4001, and a frame is acquired from the camera 1000. The acquired frame is recorded in the RAM 2003 or the recording medium 2012 and then transferred to the encoding / decoding processing unit 2013. After decoding, the frame is reproduced and displayed (video display) on the display unit 2007 via the output processing unit 2006.

  In step S4009, it is determined whether or not the start point information transmitted in step S4005 is added to the frame acquired in step S4008. If the frame at the start of recording has been received, the process proceeds to step S4010. If not, the process proceeds to step S4011.

  In step S4010, the frame and audio data acquired in step S5008 are synthesized and recording on the recording medium 2012 is started.

  In step S4011, it is determined whether or not to end the process. Otherwise, the process returns to step S4002.

  Next, processing executed by the CPU 1001 after the camera 1000 establishes connection with the terminal 2000 will be described with reference to FIG.

  In step S5001, frame generation start processing accompanying live streaming start is performed. The CPU 1001 starts signal output from the image sensor 1014, processes the output into appropriate video data by the camera signal processing unit 1015, and passes the data to the encoding / decoding processing unit 1016. The encoding / decoding processing unit 1016 starts a process of compressing and encoding the received video data at a predetermined bit rate and format format.

  In step S5002, the presence / absence of a frame acquisition request (step 3004) from the terminal 2000 is determined. If there is a frame acquisition request, the process proceeds to step S5003; otherwise, the process proceeds to step S5004.

  In step S5003, a frame is transmitted as a response (step 3005) to the frame acquisition request (step 3004).

  In step S5004, it is determined whether a start point information addition request is received from terminal 2000. The fact that the request has been received means that the self-shooting mode is being performed and the audio signal needs to be synchronized with the video signal processed by the camera signal processing unit 1015. If there is a frame acquisition request, the process proceeds to step S5005, and if not, the process proceeds to step S5006.

  In step S5005, start point information is added to the frame at the start of recording. In order to synchronize the audio signal with the video signal processed by the camera signal processing unit 1015 when the request is received in step S5004, the CPU 1001 adds a flag to the digital signal. The encoding / decoding processing unit 1016 performs predetermined processing, and adds start point information to the application marker (APPn) of the frame stored in the RAM 1003 or the recording medium 1012. The starting point information may be added as information different from the frame instead of the application marker (APPn) of the frame, and a structure combined with the frame may be defined as a response frame.

  In step S5006, it is determined whether or not to end the process, and the process returns to step S5002 except when the process is ended.

  Note that each event determination step S5002 and S5004 in this flow has been described by the determination by inquiry in sequence, but when waiting for an event of each condition at the same time, each process may be executed in the order of event occurrence.

  [Embodiment 2] Next, live streaming using a playlist will be described as Embodiment 2.

  In addition, since the system configuration of this embodiment is the same as that of Embodiment 1, the description thereof is omitted.

First, the basic operation of streaming using a playlist in the system of this embodiment will be described.
1. The camera (server) divides the streaming data into segments of a certain time, and creates a playlist that lists the segment acquisition destinations. In the case of live performance, a segment is periodically generated. Therefore, when a new segment is generated, a slide window type playlist that is dynamically updated (deleted or added) to a playlist having new contents is used.
2. The terminal (client) acquires and analyzes the playlist, and acquires data from the segment data acquisition destination in the order of enumeration.
3. The terminal (client) reproduces, displays, or saves the acquired data.
4). The camera (server) and terminal (client) repeat steps 1 to 3 until the end of the playlist (end of streaming).

  Based on the above basic operation, a live streaming operation using a playlist by the system of the present embodiment will be described with reference to FIGS.

  When the user executes the live streaming mode from the operation unit 1005 in the camera 1000, the camera 1000 sets the communication control unit 1008 in a communicable state under the control of the CPU 1001.

  Further, the user operates the operation unit 2005 on the terminal 2000 to activate an application necessary for communication connection processing and live streaming. In response to this, the CPU 2001 of the terminal 2000 controls the communication control unit 2008 according to the program stored in the ROM 2002 or the recording medium 2012, starts communication with the camera 1000, and performs connection processing.

  Here, the camera 1000 and the terminal 2000 use HTTP (Hypertext Transfer Protocol) / TCP (Transmission Control Protocol) as a communication protocol. Further, it is assumed that the communication connection supports UPnP (Universal Plug and Play). When the UPnP compatible terminal 2000 connects the device to the network, the IP (Internet Protocol) address is set by DHCP (Dynamic Host Configuration Protocol) or AutoIP. In order to recognize each other device on the network, the device that has acquired the IP address acquires information such as device search and response device type and service function by “device discovery and control” (step 6003). . In response to the device search request from the terminal 2000, the camera 1000 responds with device information, playlist acquisition destination information of device specific information, and the like (step 6004).

  When the connection process between the camera 1000 and the terminal 2000 is completed, the camera 1000 starts live streaming.

  In live streaming, the CPU 1001 of the camera 1000 starts signal output from the image sensor 1014, processes the output into appropriate video data by the camera signal processing unit 1015, and sends the data to the encoding / decoding processing unit 1016.

  The encoding / decoding processing unit 1016 compresses and encodes the received video data and the like with a predetermined bit rate and format, further divides the data with a predetermined time length Ts, and stores it as segment data (6002) in the RAM 1003 or the recording medium 1012. . In this embodiment, Ts = 0.03 seconds will be described below.

  The CPU 1001 generates path information associated with the storage destination of the segment data. The path information is used as acquisition destination information when the terminal 2000 acquires a segment, and the CPU 1001 creates a playlist 6001 and records the segment information together with the path information.

  Here, the playlist 6001 will be described with reference to FIG.

  A playlist 7010 shown in FIG. 7 is an extended M3U format playlist. An identifier tag is described in the first row 7011, and a tag and version indicating the playlist version are described in the second row 7012. In this example, the version is “3”. The third line 7013 describes a tag indicating the time of the segment data 6002 and the time (seconds) as an integer or a decimal. In this embodiment, since the segment data time length is Ts = 0.03 (seconds), it is also set to “0.03” in this example. The fourth line 7014 describes the acquisition destination path (including query parameters) of the segment data 6002. The third line 7013 and the fourth line 7014 must be described continuously as information on the segment data 6002.

  The playlist 7010 is an example of the content of the playlist 6001 in FIG. 6 in which the segment information 7013 and 7014 are recorded.

  It is assumed that the terminal 2000 stores a predetermined time length Ts of the segment data 6002 in advance or can be obtained at the time of device information acquisition by including it in the device information of the camera 1000.

  The terminal 2000 makes a playlist acquisition request (HTTP GET method) to the playlist acquisition destination acquired in step 6004 about Ts (seconds) after the start of live streaming (step 6005).

  The camera 1000 transmits a playlist 6001 (= 7010) in which one piece of segment information 7013 and 7014 is described as a response playlist (step S6006).

  The terminal 2000 analyzes the received playlist 7010 and makes a segment acquisition request (HTTP GET method) to the segment information acquisition destination (step 6007).

  The camera 1000 transmits the requested segment 6002 as a response segment (step 6008).

  The terminal 2000 passes the received segment 6002 to the encoding / decoding processing unit 2013, decodes it, and performs playback display on the display unit 2007 via the output processing unit 2006. When the user inputs an instruction to the terminal REC10007 by the live streaming application, the decoded data or the data portion excluding the header from the segment 6002 is recorded in the recording medium 2012. Then, the sequentially received segment data and the audio signal from the audio signal processing unit 2015 are encoded, multiplexed and recorded.

  During streaming, the camera 1000 generates a segment and updates a playlist about every Ts (seconds). Also, the segment information acquired by the terminal 2000 is deleted from the playlist.

  Whenever the playlist is updated by the camera 1000, the terminal 2000 obtains the playlist (step 6005) about every Ts (seconds), and based on the segment information described in the playlist 6001, the segment data Request to get.

  Note that a unique ID of the terminal or application is added to the request processing (steps 6005 and 6007) of the terminal 2000. Then, the camera 1000 of this embodiment performs streaming only for the request for the ID requested first. That is, it is assumed that the camera 1000 and the terminal 2000 of this embodiment perform streaming only with a one-to-one connection.

  Next, processing executed by the CPU 2001 after the terminal 2000 establishes connection with the camera 1000 will be described with reference to FIG.

  In step S9001, the playlist acquisition destination information is acquired from the device / playlist acquisition destination information (step 6004) acquired at the time of connection and held.

  In step S9002, it is determined whether the terminal REC10007 has been executed by the live streaming application. Note that this determination is not limited to whether the terminal REC10007 has been executed, but can be determined to start recording when the camera 1000 detects the target subject. If it has been executed, the process advances to step S9003. If it is not executed, or if it has already been executed and recording has started, the process advances to step S9013.

  In step S9003, voice input from the microphone 2014 is started.

  In step S9004, a playlist acquisition request (step 6005) is made using the playlist acquisition destination information acquired in step S9001, and the playlist acquired from the camera 1000 is analyzed. In playlist analysis, segment information is acquired after confirming the playlist format and version using an identification tag.

  In step S9005, it is determined whether segment information exists in step S9004. If the segment information exists, the process proceeds to step S9006. If the segment information does not exist, the process returns to step S9004.

  In step S9006, it is determined whether the setting of self-shooting / remote shooting 10008 is the self-shooting mode or the remote shooting mode. In the case of the self-photographing mode, the process proceeds to step S9007, and in the case of the remote-photographing mode, the process proceeds to step S9011. Note that the mode setting is set in advance by the user via the operation unit 2005. In addition, there is a terminal 2000 that does not set the mode from the operation unit 2005 and sets the self-portrait mode when the user's line of sight is detected within the angle of view of the camera 1000 or communicates within the angle of view of the camera 1000. When doing so, it is possible to set the self-portrait mode.

  In step S9007, the number of the oldest segment among the segments described for the first time in the playlist acquired in step S9004 is set as the start point number. In the self-portrait mode, it is necessary to synchronize audio data with a segment generated immediately after the terminal REC10007 is executed. For example, the playlist acquired immediately before the terminal REC10007 is executed is the playlist 8010 in FIG. 8A, and the playlist acquired immediately after the terminal REC10007 is executed is the playlist 8020 in FIG. If there is, the starting point number is 4.

  In step S9008, a segment acquisition request is made to the segment information acquisition destination path (step 6007), and the segment is acquired from the camera 1000. Here, if the acquired segment information is one, that segment is set as the oldest segment. If there are a plurality of pieces of acquired segment information, the oldest segment is set as the oldest segment. The acquired segment is recorded in the RAM 2003 or the recording medium 2012, and then transferred to the encoding / decoding processing unit 2013. After decoding, the segment is reproduced and displayed (video display) on the display unit 2007 via the output processing unit 2006.

  In step S9009, it is determined whether or not the segment number acquired in step S9008 matches the start point number set in step S9007. If they match, the process proceeds to step S9010. If they do not match, the process returns to step S9005.

  In step S9010, the segment acquired in step S9008 and the audio data are synthesized and recording on the recording medium 2012 is started.

  In step S9011, similarly to step S9008, a segment acquisition request is made to the segment information acquisition destination path (step 6007), and the segment is acquired from the camera 1000. The acquired segment is recorded in the RAM 2003 or the recording medium 2012, and then transferred to the encoding / decoding processing unit 2013. After decoding, the segment is reproduced and displayed (video display) on the display unit 2007 via the output processing unit 2006.

  In step S9012, the segment reproduced and displayed on the display unit 2007 and the audio data are combined and recording on the recording medium 2012 is started. In the case of the remote shooting mode, it is necessary to synchronize the sound with the segment displayed on the display unit 2007.

  In step S9013, it is determined whether a predetermined time has elapsed. If the predetermined time has elapsed, the process proceeds to step S9014. If the predetermined time has not elapsed, step S9002 is repeated again. The predetermined time may not be equal to the predetermined time length Ts generated by the camera 1000.

  In step S9014, similarly to step S9004, a playlist acquisition request (step S6005) is made using the playlist acquisition destination information acquired in step S9001, and the playlist acquired from the camera 1000 is analyzed.

  In step S9015, as in step S9005, it is determined whether segment information is present in step S9014. If the segment information exists, the process proceeds to step S9016. If the segment information does not exist, the process returns to step S9014.

  In step S9016, as in step S9008, a segment acquisition request (7007) is made to the segment information acquisition destination path, and the segment is acquired from the camera 1000. The acquired segment is recorded in the RAM 2003 or the recording medium 2012, and then transferred to the encoding / decoding processing unit 2013. After decoding, the segment is reproduced and displayed (video display) on the display unit 2007 via the output processing unit 2006.

  In step S9017, as in step S9013, it is determined whether a predetermined time has elapsed. If the predetermined time has elapsed, the process proceeds to step S9018; otherwise, the process returns to step S9015. If the predetermined time has not elapsed, the process returns to step S9015, and if segment information still exists in the playlist, the segment can be acquired again. The predetermined time may not be equal to the predetermined time length Ts generated by the camera 1000.

  In step S9018, it is determined whether or not to end the process. Otherwise, the process returns to step S9002.

  According to each of the embodiments described above, in the self-portrait mode, it is necessary to match the voice of the photographer's lips with the sound. Since the photographer is talking while watching the delayed video, the delayed image (video displayed on the terminal) and the sound are recorded in synchronization.

  Thus, since it is determined whether to synthesize and record the video captured by the camera and the audio input from the terminal according to the communication delay and application, it is possible to optimize the synchronization between the video and audio.

  [Other Embodiments] The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads and executes the program code. It is processing to do. In this case, the program and the storage medium storing the program constitute the present invention.

Claims (12)

A communication device capable of communicating with an imaging device,
Receiving means for receiving video data from the imaging device in a predetermined cycle;
Display means for displaying video data received by the receiving means;
Input means for inputting voice data;
Setting means for setting the shooting mode;
Recording means for synthesizing and recording the video data received by the receiving means and the audio data input by the input means;
Depending on the shooting mode set by the setting means and the delay until the video shot by the imaging device is displayed on the display means, the video data received by the receiving means and input by the input means And a control means for controlling the recording means so that the audio data is recorded synchronously. The photographing mode includes a first mode in which a photographer can see and photograph a subject without going through the display means, and a second mode in which the photographer cannot see a subject without going through the display means,
When the control unit is set to the first mode, the control unit controls the recording unit so that the video and the audio that the photographer is viewing are synchronized,
2. The communication apparatus according to claim 1, wherein in the second mode, the recording unit is controlled so that a video and an audio displayed on the display unit are synchronized. 3. When the first mode is set, information indicating that the recording device starts recording data is added to the image data transmitted from the imaging device to the imaging device. A transmission means to request from
Determining means for determining whether the information is added to the video data received by the receiving means;
3. The communication apparatus according to claim 2, wherein the control unit controls the recording unit to synthesize and record the video data to which the information is added and the audio data input by the input unit. .   When the second mode is set, the control means displays the video displayed on the display means regardless of whether or not the information is added to the video data received by the receiving means. 4. The communication apparatus according to claim 3, wherein the recording unit is controlled to synthesize and record the data and the voice data input by the input unit. First transmission means for dividing video data obtained by the imaging device into segments of a predetermined time length and transmitting a request for a list describing information of the divided segments to the imaging device;
First receiving means for receiving the requested list from the imaging device;
Second transmission means for transmitting a request for a segment described in a list received from the imaging apparatus to the imaging apparatus;
Second receiving means for receiving a segment transmitted from the imaging device in response to the request;
The first transmission means requests a list every predetermined time, and transmits a request for a segment corresponding to the oldest segment information among the segment information described in the list,
In the case of the first mode, the control means determines the oldest segment among the segments newly described in the list received by the first receiving means immediately after the recording means starts recording. The voice data input by the input means is synthesized and recorded, and in the case of the second mode, the oldest segment among the segments described in the list received by the first reception means and the communication apparatus according to any one of claims 2, wherein the controller controls the recording means to record by synthesizing the audio data inputted by the input means 4.   The said setting means sets to said 1st mode, when a photographer or the said communication apparatus exists in the angle of view of the said imaging device, The any one of Claim 2 thru | or 5 characterized by the above-mentioned. Communication equipment. An imaging device capable of communicating with a communication device,
Imaging means;
Holding means for holding video data obtained by the imaging means;
Receiving means for receiving a request for transmission of video data from the communication device;
Transmitting means for transmitting video data requested from the communication device to the communication device;
A determination unit that determines whether or not the communication device has requested to add information indicating that the video data transmitted by the transmission unit is data at the start of recording by the communication device;
An image pickup apparatus comprising: a control unit that controls the transmission unit so that the information is added to video data and transmitted to the communication device when the information is requested to be added. Holding means for holding the video data obtained by the imaging means divided into segments of a predetermined time length; and
Creating means for creating a list describing information of segments held in the holding means;
First receiving means for receiving a request for a list from the communication device;
First transmission means for transmitting the list requested from the communication device to the communication device;
Second receiving means for receiving a request for a segment described in the list from the communication device;
The imaging apparatus according to claim 7, further comprising a second transmission unit configured to transmit a segment requested by the communication apparatus to the communication apparatus. A control method for a communication device capable of communicating with an imaging device, comprising display means for displaying video data, input means for inputting audio data, and setting means for setting a shooting mode,
A receiving step of receiving video data from the imaging device in a predetermined cycle;
A display step of displaying the received video data on the display means;
A recording step of synthesizing and recording the received video data and audio data input by the input means;
In the recording step, the received video data and the input unit according to the shooting mode set by the setting unit and the delay until the video shot by the imaging device is displayed on the display unit. A method for controlling a communication device, wherein the input audio data is recorded synchronously. A method for controlling an imaging apparatus capable of communicating with a communication apparatus,
Holding the video data obtained by the imaging means;
Receiving a request for transmission of video data from the communication device;
Transmitting video data requested from the communication device to the communication device;
Determining whether or not the communication device has requested to add to the video data to be transmitted information indicating that it is data at the start of recording by the communication device;
And a step of adding the information to video data and transmitting the video data to the communication device when the information is requested to be added.   The program for functioning a computer as each means of the communication apparatus of any one of Claim 1 thru | or 6.   The program for functioning a computer as each means of the imaging device of Claim 7 or 8.

Images