JP2014057217A - Device, method and program for encoding moving image, and moving image communication device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce a delay time in moving image data communication.SOLUTION: A moving image coding device 20A encodes each of selected imaged images successively imaged by a selected camera selected from among a plurality of imaging cameras Cam0A-Cam11A and non-selected imaged images imaged by a non-selected camera other than a selected camera, among imaged images obtained by imaging an imaging object by the plurality of imaging cameras Cam0A-Cam11A, by a coding method using interframe prediction in which a frame image obtained by encoding a past selected moving image by the coding method is used as a common reference image. When the selected camera is switched to another imaging camera, the moving image coding device 20A switches the common reference image to a frame image obtained by encoding the past imaged image imaged by the other imaging camera using the coding method. The moving image coding device 20A also transmits the encoded coded data to a device of an opposite communication party.

Description

  The disclosed technology relates to a moving image encoding apparatus, a moving image encoding method, a moving image encoding program, and a moving image communication apparatus.

  With the widespread use of high-speed networks and the increased processing power of processors installed in personal computers and mobile phones, it is possible to communicate with people at long distances such as videophones and videophone conferences through video communication It has become.

  Also, in video communication, in order to pursue the reality that the communication partner is as close as possible, elements such as line-of-sight matching with the communication partner, high resolution and actual size image display, communication at a high frame rate and low delay, etc. Is very important.

  Conventionally, the position or line of sight of a communication partner is detected from a captured image obtained by capturing the communication partner with a camera, and a camera selected from a plurality of cameras that are shooting itself based on the detected position and line of sight of the communication partner is detected. There is a technique for encoding a captured image and transferring it to a communication partner. According to this technology, it is possible to eliminate the shift in the line of sight due to the movement of the person, and it is possible to obtain a sense of reality as if the user is talking through the window.

JP 2004-193962 A JP 2011-91614 A JP 2008-252651 A

  However, in the above technique, when moving image data is compressed and transmitted in order to reduce the amount of transfer data, the captured image is switched by switching the camera to be photographed after the position of the communication partner changes, Communication of moving image data is delayed by the time required for encoding.

  As a multi-view video encoding method, a method of transferring video from multiple viewpoints by performing efficient encoding using motion prediction between viewpoints is ITU-T Rec. H.264. It is standardized by Annex H (Multiview video coding, MVC).

  However, both the encoding side and the decoding side must support the MVC standard, and MPEG-2 and ITU-T Rec. H.264 (non-MVC), which are widely used as video encoding methods, are required. ) Etc. cannot be used.

  An object of the disclosed technique is to reduce the delay time of communication of moving image data, as one aspect.

  In the disclosed technology, the encoding unit is configured to select a selected captured image sequentially captured by a selected camera selected from a plurality of shooting cameras and a non-selected camera other than the selected camera from among captured images captured by a plurality of shooting cameras. Each non-selected photographed image photographed by the selected camera is encoded. The encoding unit encodes a frame image obtained by encoding a past selected captured image by an encoding method using inter-frame prediction as a common reference image. When the selected camera is switched to another shooting camera, the switching unit converts the common reference image into a frame image obtained by encoding a shot image shot by the other shooting camera in the past by the encoding method. Switch. The transmission unit transmits the encoded data encoded by the encoding unit to the apparatus on the communication partner side.

  As one aspect, the disclosed technology has an effect that the delay time of communication of moving image data can be reduced.

It is a block diagram of a moving image communication system. It is an image figure at the time of applying a moving image communication system to a videophone. It is a figure for demonstrating the stereo method. It is a block diagram of a moving image encoder. It is a figure for demonstrating a selection camera and an adjacent camera. It is a figure for demonstrating the reference relationship of the frame image which encoded the picked-up image of a selection camera and an adjacent camera. FIG. 11 is a block diagram of a computer that functions as a moving image encoding apparatus. It is a flowchart which shows the flow of a moving image encoding process. It is a flowchart which shows the flow of an input image capture process. It is a flowchart which shows the flow of an encoding process. It is a flowchart which shows the flow of a stream output process. It is an image figure at the time of applying a moving image communication system to a videophone. It is a figure for demonstrating the process timing of each process.

  Hereinafter, an example of an embodiment of the disclosed technology will be described in detail with reference to the drawings.

  FIG. 1 shows a moving image communication system 10 according to the present embodiment. The moving image communication system 10 has a configuration in which a moving image communication device 12A and a moving image communication device 12B are connected via a network 14.

  The moving image communication device 12A includes a selected camera position specifying unit 16A, detection cameras 18A1 and 18A2, a moving image encoding device 20A, a plurality (12 as an example in the present embodiment) of photographing cameras Cam0A to Cam11A, and a decoding device. 22A and a display 24A.

  The moving image communication apparatus 12B has the same configuration as that of the moving image communication apparatus 12A. The selected camera position specifying unit 16B, the detection cameras 18B1 and 18B2, the moving image encoding apparatus 20B, the shooting cameras Cam0B to Cam11B, and the decoding apparatus 22B. And a display 24B.

  In the present embodiment, a case where the moving image communication system 10 is applied to a videophone system as an example will be described. The moving image communication system 10 is not limited to a videophone system, and can be applied to any system that communicates moving images such as a video conference system.

  FIG. 2 shows an example in which the moving image communication system 10 is applied to a videophone system. In the example shown in the figure, a person 26A in the room A and a person 26B in the room B are provided on the display 24A provided on the wall with the room A and on the wall of the room B. It is possible to have a conversation while referring to the images of the opponents displayed on the display 24B. In FIG. 2, for convenience, the room A and the room B are described as being adjacent to each other. However, the room A and the room B are actually separated from each other.

  On the wall surface of the room A, twelve photographing cameras Cam0A to Cam11A are attached in a lattice shape, and each photographing camera photographs a person 26A in the room A. Also, twelve photographing cameras Cam0B to Cam11B are attached to the wall surface of the room B in the same arrangement as the photographing cameras Cam0A to Cam11A provided on the wall surface of the room A. The photographing cameras Cam0B to Cam11B photograph the person 26B in the room B.

  In the room A, detection cameras 18A1 and 18A2 for detecting the position of the person 26A are provided. The detection cameras 18A1 and 18A2 are provided at positions for photographing the person 26A in the room A from different directions.

  Similarly, the room B is provided with detection cameras 18B1 and 18B2 for detecting the position of the person 26B. The detection cameras 18B1 and 18B2 are provided at positions for photographing the person 26B in the room B from different directions.

  The selected camera position specifying unit 16A calculates, for example, a three-dimensional position of the face of the person 26A based on the captured image of the person 26A captured by the detection cameras 18A1 and 18A2. Then, the position of the photographing camera provided in the room B corresponding to the photographing camera closest to the calculated position of the face of the person 26A is specified as the selected camera position, and is transmitted to the moving picture coding apparatus 20B. For example, when the photographing camera closest to the calculated face position of the person 26A is the photographing camera Cam5A, the position of the photographing camera provided on the wall surface of the room B at the position corresponding to the photographing camera Cam5A is specified as the selected camera position. And transmitted to the moving image communication apparatus 12B.

  Specifically, the selected camera position specifying unit 16A calculates the three-dimensional position of the face of the person 26A using the stereo method. The stereo method will be briefly described below.

  As shown in FIG. 3, a photographing target point P in a three-dimensional space photographed from the viewpoint A of the camera is observed as a point i where the straight line LA connecting the viewpoint A and the point P and the photographing plane DA intersect. Similarly, the photographing target point P in the three-dimensional space photographed from the viewpoint B of the camera is observed as a point j where the straight line LB connecting the viewpoint B and the point P and the photographing plane DB intersect. Therefore, the shooting target point P is observed as points i and j in the two shooting planes DA and DB, respectively, and the straight line LA connecting the viewpoint A and the point i and the straight line LB connecting the viewpoint B and the point j are as follows. Intersect at point P. That is, when the positional relationship between the viewpoints A and B is known, the three-dimensional position of the shooting target point P in the three-dimensional space is uniquely determined by specifying the positions on the two shooting planes DA and DB with respect to the shooting target point P. Determined.

  Therefore, first, the position of the facial feature point of the person 26A is referred to as a so-called SIFT (Scale Invariant Feature Transform), or SURF (Speed Up Robust Features) that speeds up SIFT, etc. Each of these is calculated using the above method. Next, the three-dimensional position of the face of the person 26A is calculated from the position of the face of the person 26A in each captured image. Then, the camera that is closest to the calculated face position of the person 26A is specified, and the position of the camera provided in the room B corresponding to the specified camera is transmitted to the moving image communication apparatus 12B as the selected camera position.

  Similarly, the selected camera position specifying unit 16B of the moving image communication apparatus 12B calculates the three-dimensional position of the face of the person 26B based on the captured image of the person 26B captured by the detection cameras 18B1 and 18B2. Then, the position of the shooting camera provided in the room A corresponding to the shooting camera closest to the calculated position of the face of the person 26B is specified as the selected camera position, and is transmitted to the moving picture coding apparatus 20A. Therefore, for example, as shown in FIG. 2, when the person 26B in the room B is closest to the shooting camera Cam6B, the selected camera position specifying unit 16B specifies the shooting camera Cam6A corresponding to the shooting camera Cam6B as the selected camera position to make a moving image. It transmits to the image communication apparatus 12A.

  The moving image encoding device 20 </ b> A has captured a captured image of the person 26 </ b> A captured by the selected camera selected based on the position of the person 26 </ b> B in the room B among the plurality of capturing cameras Cam <b> 0 </ b> A to Cam <b> 11 </ b> A. A past photographed image is encoded as a common reference image. As an encoding method, for example, an encoding method using inter-frame prediction such as MPEG-2 or H.264 (non-MVC) can be used, and a multi-view encoding method such as H.264 (MVC) can be used. There is no need to use it.

  In addition, the moving image encoding device 20A shares a captured image captured by a non-selected camera other than the selected camera selected based on the position of the person 26B in the room B with a past captured image captured by the selected camera. Use as a reference image for encoding.

  In addition, when the selected camera is switched, the moving image encoding device 20A switches the common reference image to a captured image captured by the switched selected camera.

  FIG. 4 shows a specific configuration of the moving image encoding device 20A. The moving image encoding device 20A includes an overall control unit 30, an input image capturing unit 32, an encoding unit 34, a stream output unit 36, a memory controller 38, and an SDRAM 40.

  The input image capturing unit 32 includes a camera selection unit 42 and a selected camera selector 44X, an adjacent camera A selector 44A to an adjacent camera K selector 44K, a selected camera image input unit 46X, and an adjacent camera A image input unit 46A to an adjacent camera K image input. 46K is provided.

  The encoding unit 34 includes a camera selection unit 48, a selected camera image encoding unit 52X, and an adjacent camera A encoding unit 52A to an adjacent camera K encoding unit 52K.

  The stream output unit 36 includes a camera selection unit 54 and a stream selection unit 56.

  As illustrated in FIG. 4, the overall control unit 30 outputs the selected camera position transmitted from the moving image encoding device 20 </ b> B to the camera selection units 42, 48, and 54. In addition, when the user instructs the start of moving image communication, the overall control unit 30 instructs the camera selection unit 42 to start capturing captured images.

  The camera selection unit 42 instructs the selected camera selector 44X to select a photographing camera corresponding to the selected camera position input from the overall control unit 30 as the selected camera. Shooting cameras Cam0A to Cam11A are connected to the selected camera selector 44X.

  The selected camera selector 44X selects a shooting camera corresponding to the selected camera position designated by the camera selection unit 42 from the shooting cameras Cam0A to Cam11A, and outputs a shot image of the selected shooting camera to the selection camera image input unit 46X. To do.

  In addition, the camera selection unit 42 instructs each of the adjacent camera A selector 44A to the adjacent camera K selector 44K to select each adjacent camera using the shooting camera adjacent to the shooting camera corresponding to the selected camera position as the adjacent camera. To do. Shooting cameras Cam0A to Cam11A are connected to each of the adjacent camera A selector 44A to the adjacent camera K selector 44K.

  For example, when the shooting cameras are provided in a grid pattern as shown in FIG. 2, four shooting cameras positioned on the top, bottom, left, and right of the selected camera can be set as the adjacent cameras A to D. In this case, the camera selection unit 42 instructs each of the adjacent camera A selector 44A to the adjacent camera D selector 44D to select a photographing camera adjacent to the selected camera. Hereinafter, a case where four photographing cameras adjacent to the selected camera in the vertical and horizontal directions are set as the adjacent cameras A to D will be described.

  The adjacent camera A selector 44A selects a shooting camera corresponding to the adjacent camera A instructed by the camera selection unit 42 from the shooting cameras Cam0A to Cam11A, and sets the captured image of the selected shooting camera to the adjacent camera A image input unit 46A. Output to. The same applies to the adjacent camera B selector 44B to the adjacent camera D selector 44D.

  Therefore, in this embodiment, the eleven adjacent camera A selector 44A to the adjacent camera K selector 44K are provided, but selectors may be provided as many as the number of adjacent cameras. The same applies to the adjacent camera A image input unit 46A to the adjacent camera K image input unit 46K, and as many image input units as the number of adjacent cameras may be provided. Furthermore, each storage area related to the adjacent camera of the SDRAM 40 to be described later may be provided by the number of adjacent cameras.

  The selected camera image input unit 46 </ b> X performs a predetermined filter process or the like on the input captured image and outputs the result to the memory controller 38. The same applies to the adjacent camera A image input unit 46A to the adjacent camera K image input unit 46K.

  For example, when the photographing camera Cam6A is selected as the selected camera, the overall control unit 30 instructs the selected camera selector 44X to select the photographing camera Cam6A. As a result, the selected camera selector 44X outputs the captured image of the imaging camera Cam6A to the selected camera image input unit 46X.

  Further, in the state before the person 26B in the room B shown in FIG. 5 moves, when the shooting camera Cam6A is selected as the selection camera, the shooting cameras Cam5A, Cam7A, Cam2A, and Cam10A adjacent to the left and right and up and down of the shooting camera Cam6A are displayed. Let it be adjacent cameras A to D. Then, the camera selection unit 42 instructs the adjacent camera A selector 44A to select the photographing camera Cam5A. Thereby, the adjacent camera A selector 44A outputs the captured image of the shooting camera Cam5A to the adjacent camera A image input unit 46A.

  Similarly, the camera selection unit 42 instructs the adjacent camera B selector 44B to select the photographing camera Cam7A. Thereby, the adjacent camera A selector 44B outputs the captured image of the shooting camera Cam7A to the adjacent camera B image input unit 46B. The same applies to the adjacent cameras C and D.

  The memory controller 38 is connected to the selected camera image input unit 46X and the adjacent camera A image input unit 46A to the adjacent camera K image input unit 46K. The memory controller 38 stores the captured image input from the selected camera image input unit 46X as a selected camera original image in the selected camera original image storage area 40X1 of the SDRAM. Further, the memory controller 38 stores the captured image input from the adjacent camera A image input unit 46A as the adjacent camera A original image in the adjacent camera A original image storage area 40A1 of the SDRAM. The same applies to the adjacent cameras B to D.

  When an instruction to start the encoding process is given from the overall control unit 30, the selected camera image encoding unit 52X stores the selected camera original image stored in the selected camera original image storage area 40X1 and the selected camera reference image area 40X2. The captured image is encoded based on the common reference image. The common reference image will be described later.

  As an encoding method, for example, an encoding method using inter-frame prediction such as MPEG-2, MPEG-4, H.264 (non-MVC) or the like is used. In this embodiment, an IPPP structure is used as a GOP (Group Of Pictures) structure. That is, a structure using an I picture that can be decoded independently and a P picture in which a difference is encoded using a past frame image as a reference image. In this way, since the B picture that is encoded using not only the past frame image but also the future frame image as the reference image is not used, the time required for encoding can be shortened.

  Further, the selected camera image encoding unit 52X generates bit stream data including encoded data, motion vectors, and the like and stores them in the selected camera stream storage area 40X3 of the SDRAM 40.

  Similar to the selected camera image encoding unit 52X, the adjacent camera A encoding unit 52A to the adjacent camera D encoding unit 52D are also stored in the corresponding reference image region and the selected camera original image stored in the corresponding original image region. The captured image is encoded based on the common reference image. Then, bit stream data is generated and stored in the corresponding stream storage area of the SDRAM 40.

  FIG. 6 shows an example of a frame image obtained by encoding images taken by the selected camera X and the adjacent cameras A to D. As shown in the drawing, the frame image 60X0 of the selected camera X and the frame images 60A0 to 60D0 of the adjacent cameras A to D encoded at the timing of the picture 0 are I pictures. Thereafter, the frame images of the selected camera X and the adjacent cameras A to D from the pictures 1 to 6 are all P pictures.

  At the timing of picture 0, the selected camera X is the shooting camera Cam6A, the adjacent camera A is the shooting camera Cam5A, the adjacent camera B is the shooting camera Cam7A, the adjacent camera C is the shooting camera Cam2A, and the adjacent camera D is the shooting camera Cam10A.

  At the timing of the next picture 1, the frame image 60X1 of the selected camera X is a P picture, and the frame image 60X0 (I picture) of the selected camera X encoded at the timing of picture 0 is encoded as a common reference image.

  The frame images 60A1 to 60D1 of the adjacent cameras A to D at the timing of the picture 1 are P pictures, and the frame image 60X0 of the selected camera X encoded at the timing of the picture 0 is encoded as a common reference image.

  As described above, at the timing of the picture 1, the frame image 60X0 of the selected camera X encoded at the timing of the immediately preceding picture 0 is used as the common reference image for all the images taken by the selected camera X and the adjacent cameras A to D. Is encoded as

  Similarly, at the timing of picture 2, the frame image 60X1 of the selected camera X encoded at the timing of the immediately previous picture 1 is used as a common reference image for all images taken by the selected camera X and adjacent cameras A to D. Encode.

  Thereafter, until the selected camera X is switched to another photographing camera, the frame image of the selected camera X encoded at the immediately preceding timing is used as a common reference image, and the captured images of the selected camera X and the adjacent cameras A to D are encoded. Is done.

  In the example of FIG. 6, since the selected camera X is switched from the shooting camera Cam6 to the shooting camera Cam5 at the time of the picture 3, the frame image obtained by encoding the shot image of the shooting camera Cam6 is the common reference image up to the picture 3. .

  When the selected camera X is switched from the photographic camera Cam6 to the photographic camera Cam5 at the time of the picture 3, after the picture 4, the selected camera X is the photographic camera Cam5, and the adjacent cameras A to D are the photographic cameras Cam4, Cam6, Cam1 and Cam9.

  Since the selected camera X is switched to the photographing camera Cam5 at the timing of the picture 4, the frame image 60A3 obtained by encoding the photographed image of the photographing camera Cam5 which is the adjacent camera A at the timing of the immediately preceding picture 3 is shared. Let it be a reference image.

  Since the selected camera X is set to the photographic camera Cam5 after the picture 5, a frame image obtained by encoding the photographic image taken by the selected camera X at the immediately preceding timing is used as a common reference image. Thus, when the selected camera X is switched, the common reference image is also switched.

  As shown in FIG. 4, the overall control unit 30 stores the immediately preceding selected camera position at the immediately preceding encoding timing in the immediately preceding selected camera position storage area 30A, and is updated every time the encoding timing arrives.

  The camera selection unit 48 compares the input current selected camera position with the immediately preceding selected camera position stored in the immediately preceding selected camera position storage area 30A. If the input current selected camera position is the same as the previous selected camera position, a frame image obtained by encoding the captured image of the selected camera X is set as a common reference image.

  In this case, the memory controller 38 stores the frame image (local decoded image) encoded by the selected camera image encoding unit 52X in the selected camera reference image area 40X2 of the SDRAM 40 as a common reference image. Further, the memory controller 38 stores the frame image encoded by the selected camera image encoding unit 52X in the adjacent camera A reference image area 40A2 to the adjacent camera D reference image area 40D2 of the SDRAM 40 as a common reference image.

  On the other hand, when the input current selected camera position is not the same as the immediately preceding selected camera position, that is, when the selected camera X is switched, a frame image obtained by encoding the captured image captured at the immediately preceding timing by the switched capturing camera. Is set as a common reference image. The camera selection unit 48 is an example of a switching unit in the disclosed technology.

  In this case, the memory controller 38 uses the frame image encoded by the encoding unit corresponding to the switched photographing camera among the adjacent camera A encoding unit 52A to the adjacent camera D encoding unit 52D as a common reference image. Are stored in each reference image area. That is, the selected camera reference image area 40X2 and the adjacent camera A reference image area 40A2 to the adjacent camera D reference image area 40D2 of the SDRAM 40 are stored.

  The camera selection unit 54 compares the input current selected camera position with the immediately preceding selected camera position. If the input current selected camera position is the same as the previous selected camera position, the stream selection unit 56 is instructed to select and output the bit stream data of the selected camera X.

  Thereby, the stream selection unit 56 reads the bit stream data from the selected camera stream storage area 40X3 of the SDRAM 40, and transmits it to the moving image communication apparatus 12B as an output stream. On the other hand, if the input current selected camera position is not the same as the previous selected camera position, that is, if the selected camera X is switched, the bit stream data is read from the stream storage area corresponding to the switched camera. And it transmits to the moving image communication apparatus 12B as an output stream. The stream selection unit 56 is an example of an encoded data transmission unit in the disclosed technology.

  The decoding device 22A decodes the bit stream data transmitted from the video communication device 12B and displays it on the display 24A.

  The moving image encoding apparatus 20A can be realized by a computer 70 shown in FIG. 7, for example. The computer 70 includes a CPU 72, a memory 74, and a nonvolatile storage unit 76, which are connected to each other via a bus 78.

  The storage unit 76 can be realized by an HDD (Hard Disk Drive), a flash memory, or the like. The storage unit 76 as a recording medium stores a moving image encoding program 80 for causing the computer 70 to function as the moving image encoding device 20A. The CPU 72 reads out the moving image encoding program 80 from the storage unit 76 and expands it in the memory 74, and sequentially executes processes included in the moving image encoding program 80.

  The moving image encoding program 80 includes an input image capturing process 82, an encoding process 84, and a stream output process 86.

  The CPU 72 operates as the input image capturing unit 32 illustrated in FIG. 4 by executing the input image capturing process 82. The CPU 72 operates as the encoding unit 34 illustrated in FIG. 4 by executing the encoding process 84. The CPU 72 operates as the stream output unit 36 illustrated in FIG. 4 by executing the stream output process 86.

  As a result, the computer 70 that has executed the moving picture coding program 80 functions as the moving picture coding apparatus 20A. The moving image encoding program 80 is an example of a moving image encoding program in the disclosed technology.

  Note that the moving picture coding apparatus 20A can be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit) or the like.

  Since the configuration of the moving image communication device 12B is the same as that of the moving image communication device 12A, description thereof is omitted.

  Next, the operation of this embodiment will be described. In the moving image encoding device 20A according to the present embodiment, when the user instructs to execute moving image communication, the moving image encoding process shown in FIG. 8 is executed. In the present embodiment, as shown in FIG. 5, a case will be described in which cameras adjacent to the selected camera X in the vertical and horizontal directions are set as adjacent cameras A to D.

  In step 100, the input image capturing unit 32 executes an input image capturing process as shown in FIG.

  In step 102, the encoding unit 34 executes an encoding process as shown in FIG.

  In step 104, the stream output unit 36 executes a stream output process as shown in FIG.

  In step 106, the overall control unit 30 determines whether or not the user has instructed the end of the moving image communication. When the end of the moving image communication is instructed, this routine is ended and the moving image communication is ended. Is not instructed, the process returns to step 100 and the above processing is repeated.

  In step 200 of FIG. 9, the camera selection unit 42 instructs the selected camera selector 44 </ b> X to select a shooting camera corresponding to the selected camera position input from the overall control unit 30. Accordingly, the selected camera selector 44X selects a photographing camera corresponding to the selected camera position instructed by the camera selection unit 42 among the photographing cameras Cam0A to Cam11A.

  In addition, the camera selection unit 42 instructs each of the adjacent camera A selector 44A to the adjacent camera D selector 44D to select each adjacent camera using the shooting camera adjacent to the shooting camera corresponding to the selected camera position as the adjacent camera. To do. As a result, the adjacent camera A selector 44A to the adjacent camera D selector 44D each select a photographing camera corresponding to the adjacent camera instructed by the camera selection unit 42.

  In step 202, when the camera selection unit 42 is instructed to start capturing captured images from the overall control unit 30, it instructs each selector to start capturing captured images. As a result, the selected camera selector 44X outputs the captured image of the selected imaging camera to the selected camera image input unit 46X. The selected camera image input unit 46 </ b> X performs a predetermined filter process or the like on the input captured image and outputs the result to the memory controller 38. The memory controller 38 stores the input photographed image of the selected camera X in the selected camera original image storage area 40X1 of the SDRAM 40.

  The adjacent camera A selector 44A to the adjacent camera D selector 44D output the captured images of the selected shooting cameras to the adjacent camera A image input unit 46A to the adjacent camera D image input unit 46D, respectively. The adjacent camera A image input unit 46 </ b> A to the adjacent camera D image input unit 46 </ b> D perform predetermined filter processing or the like on the input captured image, and output the result to the memory controller 38. The memory controller 38 stores the input captured images of the adjacent cameras A to D in the corresponding original image areas of the SDRAM 40, respectively.

  In step 204, each image input unit determines whether or not capturing of one frame of the captured image has been completed. When capturing is complete, this routine ends. When capturing is not complete, Continue to capture captured images.

  In the case of the example in FIG. 6, the selected camera is switched at the timing of picture 3. Specifically, as shown in FIG. 12, the person 26B in the room B moves, and the selected camera is switched from the photographing camera Cam6A to Cam5A. In this case, as shown in FIG. 5, the adjacent cameras A to D are also switched to the photographing cameras Cam04A, Cam6A, Cam1A, and Cam9A, respectively. For this reason, as shown in FIG. 13, in the input image capturing process, the captured images of the capturing camera Cam6A are captured as the selected camera X for the pictures 0 to 3. In addition, the captured image of the photographic camera Cam5A as the adjacent camera A, the captured image of the photographic camera Cam7A as the adjacent camera B, the captured image of the photographic camera Cam2A as the adjacent camera C, and the captured image of the photographic camera Cam10A as the adjacent camera D are captured. It is.

  As shown in FIG. 13, in the pictures 4 to 6 after the selected camera is switched, the captured image of the imaging camera Cam5A is captured as the selected camera X. Further, the captured image of the photographic camera Cam4A is captured as the adjacent camera A, the captured image of the photographic camera Cam6A as the adjacent camera B, the captured image of the photographic camera Cam1A as the adjacent camera C, and the captured image of the photographic camera Cam9A as the adjacent camera D is captured. It is.

  In step 300 of FIG. 10, the camera selection unit 48 compares the input current selected camera position with the immediately preceding selected camera position. That is, it is determined whether the selected camera has been switched. If the input current selected camera position is not the same as the previous selected camera position, that is, if the selected camera is switched, the process proceeds to step 302. On the other hand, if the input current selected camera position is the same as the previous selected camera position, that is, if the selected camera has not been switched, the process proceeds to step 304.

  In step 302, the camera selection unit 48 sets, as a common reference image, a frame image obtained by encoding a photographed image photographed at the immediately preceding timing by the selected camera after switching.

  On the other hand, in step 304, since the selected camera X has not been switched, the camera selection unit 48 continues to set a frame image obtained by encoding the captured image of the selected camera X as a common reference image.

  In step 306, the overall control unit 30 updates the last selected camera position. If the selected camera X has not been switched, the immediately preceding selected camera position is overwritten with the same value.

  In step 308, the overall control unit 30 instructs each encoding unit to start encoding. Thus, the selected camera image encoding unit 52X encodes the captured image based on the selected camera original image stored in the selected camera original image storage area 40X1 and the common reference image stored in the selected camera reference image area 40X2. Turn into.

  Further, the selected camera image encoding unit 52X generates bit stream data including encoded data, motion vectors, and the like and stores them in the selected camera stream storage area 40X3 of the SDRAM 40.

  Similarly to the selected camera image encoding unit 52X, the adjacent camera A encoding unit 52A to the adjacent camera D encoding unit 52D also select the selected camera original image stored in the corresponding original image region and the corresponding reference image region. A captured image is encoded based on the stored common reference image. Then, bit stream data is generated and stored in the corresponding stream storage area of the SDRAM 40.

  In step 310, each encoding unit determines whether or not encoding of the captured image for one frame is completed. When encoding is completed, this routine is terminated, and when encoding is not completed. Continues the encoding process.

  Accordingly, as illustrated in FIG. 13, in the encoding process, the common reference image is switched from the frame image of the photographing camera Cam6A to the frame image of the photographing camera Cam5A at the time of the picture 3.

  In step 400 of FIG. 11, the camera selection unit 54 compares the input current selected camera position with the immediately preceding selected camera position. That is, it is determined whether the selected camera has been switched. If the input current selected camera position is not the same as the previous selected camera position, that is, if the selected camera is obtained, the process proceeds to step 402. On the other hand, if the input current selected camera position is the same as the previous selected camera position, that is, if the selected camera has not been obtained, the process proceeds to step 404.

  In step 402, the camera selection unit 54 instructs the stream selection unit 56 to select the bit stream data of the photographing camera after switching as an output stream.

  On the other hand, in step 404, the camera selection unit 54 instructs the stream selection unit 56 to continue to select the bit stream data of the selected camera X as an output stream.

  In step 406, the overall control unit 30 instructs the stream selection unit 56 to start outputting the bit stream data. Thereby, the stream selection unit 56 reads the bit stream data from the selected camera stream storage area 40X3 of the SDRAM 40, and transmits it to the moving image communication apparatus 12B as an output stream.

  In step 408, the stream selection unit 56 determines whether or not the output of the stream data for one frame has been completed. If the output has been completed, this routine is terminated. If the output has not been completed, the output is output. continue.

  Accordingly, as shown in FIG. 13, in the stream output process, the output stream of the selected camera X is switched from the stream data of the photographing camera Cam5A to the stream data of Cam5A at the time of the picture 3.

  As described above, in this embodiment, not only a captured image captured by the selected camera but also a captured image of an adjacent camera adjacent to the selected camera is encoded. For this reason, the delay time of communication of moving image data can be reduced as compared with the case where imaging is started and encoded after the selected camera is switched. In addition, since the reference image when encoding captured images captured by a plurality of imaging cameras is set to a frame image that is an encoded captured image previously captured by the selected camera, the selected camera It is possible to prevent a sense of incongruity from occurring when switching. Further, there is no need to use a multi-view coding method.

  In the present embodiment, the case where twelve photographing cameras are provided in a lattice shape has been described. However, the number and arrangement of the photographing cameras are not limited to this.

  Further, in the present embodiment, the case has been described in which four shooting cameras on the top, bottom, left, and right of the selected camera are set as the adjacent cameras A to D, but the number and positions of the adjacent cameras that can be set are not limited thereto. . For example, a shooting camera positioned in the oblique direction of the selected camera may be set as the adjacent camera.

  In this embodiment, the case where the selected camera is switched according to the position of the other party has been described. However, the present invention is not limited to this, and the user may instruct switching of the selected camera.

  In the present embodiment, the case where the subject is a person is described. However, the present invention is not limited to this, and an animal other than a human may be used.

  Moreover, although the moving image encoding program 80 which is an example of the moving image encoding program which concerns on the technique of the indication demonstrated above the aspect pre-stored (installed) in the memory | storage part 76, it is not limited to this Absent. The image processing program according to the disclosed technology can be provided in a form recorded on a recording medium such as a CD-ROM or a DVD-ROM.

  All documents, patent applications and technical standards mentioned in this specification are to the same extent as if each individual document, patent application and technical standard were specifically and individually stated to be incorporated by reference. Incorporated by reference in the book.

  Regarding the above embodiment, the following additional notes are disclosed.

(Appendix 1)
Of the captured images captured by the plurality of shooting cameras, the selected shooting images sequentially shot by the selected camera selected from the plurality of shooting cameras and the non-selected cameras other than the selected camera were shot. An encoding unit that encodes each non-selected captured image by the encoding method using a frame image obtained by encoding the past selected captured image by an encoding method using inter-frame prediction as a common reference image;
A switching unit that switches the common reference image to a frame image obtained by encoding the captured image captured by the other imaging camera in the past by the encoding method when the selected camera is switched to another imaging camera; ,
An encoded data transmission unit that transmits the encoded data encoded by the encoding unit to a communication partner side device;
A video encoding apparatus including:

(Appendix 2)
The moving image coding apparatus according to claim 1, wherein the non-selected camera is a photographing camera adjacent to the selected camera among the plurality of photographing cameras.

(Appendix 3)
A detection camera for detecting the position of the photographing target;
Photographing for photographing a photographing target on the communication partner side from among a plurality of photographing cameras provided on the communication partner side based on a position of the photographing target calculated based on a photographed image photographed by the detection camera. A selected camera position specifying unit that specifies the position of the camera as the communication partner selected camera position;
A selected camera position transmitter for transmitting the communication partner selection camera position to the communication partner side device;
The moving picture encoding device according to Supplementary Note 1 or Supplementary Note 2, comprising:

(Appendix 4)
The communication partner side device is:
A communication partner side detection camera for detecting the position of the photographing target on the communication partner side;
Based on the position of the photographing target on the communication partner side calculated based on the photographed image photographed by the communication partner side detection camera, the position of the photographing camera for photographing the photographing target from the plurality of photographing cameras is determined. A communication partner side selected camera position identifying unit that identifies the selected camera position;
A communication partner-side selected camera position transmitting unit that transmits the selected camera position to the moving image encoding device;
With
The moving image encoding apparatus according to any one of appendices 1 to 3, wherein the selected camera is a photographing camera corresponding to the selected camera position transmitted from the communication partner side selected camera position transmitting unit.

(Appendix 5)
Among the photographed images of the subject photographed sequentially by a plurality of photographing cameras, a selected photographed image sequentially photographed by a selected camera selected from the plurality of photographing cameras and a non-selected photographed by a non-selected camera other than the selected camera Each of the captured images is encoded by the encoding method as a common reference image, a frame image obtained by encoding the past selected captured image by an encoding method using inter-frame prediction,
When the selected camera is switched to another shooting camera, the common reference image is switched to a frame image obtained by encoding a captured image shot by the other shooting camera in the past by the encoding method. Video encoding method.

(Appendix 6)
The moving image encoding method according to claim 3, wherein the non-selected camera is a shooting camera adjacent to the selected camera among the plurality of shooting cameras.

(Appendix 7)
Based on the position of the photographing target calculated based on a photographed image photographed by a detection camera for detecting the position of the photographing target, the communication is performed from among a plurality of photographing cameras provided on the communication partner side. Specify the position of the shooting camera that shoots the other party's shooting target as the communication partner selection camera position,
The moving image encoding method according to claim 5 or 6, wherein the communication partner selection camera position is transmitted to the device on the communication partner side.

(Appendix 8)
The communication partner side device is:
A communication partner side detection camera for detecting the position of the photographing target on the communication partner side;
Based on the position of the photographing target on the communication partner side calculated based on the photographed image photographed by the communication partner side detection camera, the position of the photographing camera for photographing the photographing target from the plurality of photographing cameras is determined. A communication partner side selected camera position identifying unit that identifies the selected camera position;
A communication partner-side selected camera position transmitting unit that transmits the selected camera position to the moving image encoding device;
With
The moving image encoding method according to any one of claims 5 to 7, wherein the selected camera is a shooting camera corresponding to the selected camera position transmitted from the communication partner side selected camera position transmitting unit.

(Appendix 9)
On the computer,
Among the photographed images of the subject photographed sequentially by a plurality of photographing cameras, a selected photographed image sequentially photographed by a selected camera selected from the plurality of photographing cameras and a non-selected photographed by a non-selected camera other than the selected camera Each of the captured images is encoded by the encoding method as a common reference image, a frame image obtained by encoding the past selected captured image by an encoding method using inter-frame prediction,
When the selected camera is switched to another shooting camera, the common reference image is switched to a frame image obtained by encoding a captured image shot by the other shooting camera in the past by the encoding method. A moving image encoding program for executing processing.

(Appendix 10)
Of the captured images captured by the plurality of shooting cameras, the selected shooting images sequentially shot by the selected camera selected from the plurality of shooting cameras and the non-selected cameras other than the selected camera were shot. An encoding unit that encodes each non-selected captured image by the encoding method using a frame image obtained by encoding the past selected captured image by an encoding method using inter-frame prediction as a common reference image; and the selection A switching unit that switches the common reference image to a frame image obtained by encoding the captured image captured by the other imaging camera in the past by the encoding method when the camera is switched to another imaging camera; A moving picture coding apparatus comprising: a coded data transmission unit configured to send coded data encoded by the coding unit to a communication partner side device ,
A decoding unit that receives and decodes the encoded data transmitted from the device on the communication partner side;
A display unit for displaying a moving image based on the decoded data decoded by the decoding unit;
A moving image communication apparatus including:

DESCRIPTION OF SYMBOLS 10 Video communication system 12A, 12B Video communication apparatus 14 Network 16A, 16B Selected camera position specific | specification part 18A1, 18A2, 18B1, 18B2 Detection camera 20A, 20B Video encoding apparatus 22A, 22B Decoding apparatus 24A, 24B Display 30 Overall control unit 32 Input image capturing unit 34 Encoding unit 36 Stream output unit 38 Memory controller

Claims (7)

Of the captured images captured by the plurality of shooting cameras, the selected shooting images sequentially shot by the selected camera selected from the plurality of shooting cameras and the non-selected cameras other than the selected camera were shot. An encoding unit that encodes each non-selected captured image by the encoding method using a frame image obtained by encoding the past selected captured image by an encoding method using inter-frame prediction as a common reference image;
A switching unit that switches the common reference image to a frame image obtained by encoding the captured image captured by the other imaging camera in the past by the encoding method when the selected camera is switched to another imaging camera; ,
An encoded data transmission unit that transmits the encoded data encoded by the encoding unit to a communication partner side device;
A video encoding apparatus including: The moving image encoding apparatus according to claim 1, wherein the non-selected camera is a photographic camera adjacent to the selected camera among the plurality of photographic cameras. A detection camera for detecting the position of the photographing target;
Photographing for photographing a photographing target on the communication partner side from among a plurality of photographing cameras provided on the communication partner side based on a position of the photographing target calculated based on a photographed image photographed by the detection camera. A selected camera position specifying unit that specifies the position of the camera as the communication partner selected camera position;
A selected camera position transmitter for transmitting the communication partner selection camera position to the communication partner side device;
The moving picture coding apparatus according to claim 1 or 2, further comprising: The communication partner side device is:
A communication partner side detection camera for detecting the position of the photographing target on the communication partner side;
Based on the position of the photographing target on the communication partner side calculated based on the photographed image photographed by the communication partner side detection camera, the position of the photographing camera for photographing the photographing target from the plurality of photographing cameras is determined. A communication partner side selected camera position identifying unit that identifies the selected camera position;
A communication partner-side selected camera position transmitting unit that transmits the selected camera position to the moving image encoding device;
With
The moving image encoding apparatus according to claim 1, wherein the selected camera is a shooting camera corresponding to the selected camera position transmitted from the communication partner side selected camera position transmitting unit. Of the captured images captured by the plurality of shooting cameras, the selected shooting images sequentially shot by the selected camera selected from the plurality of shooting cameras and the non-selected cameras other than the selected camera were shot. Each non-selected captured image is encoded by the encoding method as a common reference image, a frame image obtained by encoding the past selected captured image by an encoding method using inter-frame prediction,
When the selected camera is switched to another shooting camera, the common reference image is switched to a frame image obtained by encoding the shot image shot by the other shooting camera in the past by the encoding method,
A moving image encoding method comprising: transmitting encoded data that has been encoded to a device on a communication partner side. On the computer,
Of the captured images captured by the plurality of shooting cameras, the selected shooting images sequentially shot by the selected camera selected from the plurality of shooting cameras and the non-selected cameras other than the selected camera were shot. Each non-selected captured image is encoded by the encoding method as a common reference image, a frame image obtained by encoding the past selected captured image by an encoding method using inter-frame prediction,
When the selected camera is switched to another shooting camera, the common reference image is switched to a frame image obtained by encoding the shot image shot by the other shooting camera in the past by the encoding method,
A moving picture encoding program for executing processing including transmitting encoded encoded data to a communication partner apparatus. Of the captured images captured by the plurality of shooting cameras, the selected shooting images sequentially shot by the selected camera selected from the plurality of shooting cameras and the non-selected cameras other than the selected camera were shot. An encoding unit that encodes each non-selected captured image by the encoding method using a frame image obtained by encoding the past selected captured image by an encoding method using inter-frame prediction as a common reference image; and the selection A switching unit that switches the common reference image to a frame image obtained by encoding the captured image captured by the other imaging camera in the past by the encoding method when the camera is switched to another imaging camera; A moving picture coding apparatus comprising: a coded data transmission unit configured to send coded data encoded by the coding unit to a communication partner side device ,
A decoding unit that receives and decodes the encoded data transmitted from the device on the communication partner side;
A display unit for displaying a moving image based on the decoded data decoded by the decoding unit;
A moving image communication apparatus including: