JP2023026844A - Video recording system, video recording playback device and image processing method - Google Patents

Abstract

To provide a video recording system which can display a first image during playback even when video recording is performed in such a state that a second image is superimposed on the first image.SOLUTION: The present invention provides a video recording system 60 comprising: a communication unit which receives first image data from another base; an image processing unit which moves the position of the first image data or second image data within a frame of a video in which the second image data received from the other base or prepared in the own base is embedded in the first image; an image composition unit which generates a video file in which the second image data is embedded in the first image data; an image construction unit which constructs the first image data that does not include the second image data from the video file; and a display control unit which displays the first image data constructed by the image construction unit.SELECTED DRAWING: Figure 1

Description

The present invention relates to a recording system, a recording/playback apparatus, and an image processing method.

2. Description of the Related Art A communication system is known that transmits and receives video and audio to and from a remote location via a network. In a communication system, at a location where one of the parties participating in a conference is located, a communication terminal is used to capture images and collect voices such as remarks, which are then converted into digital data and transmitted to the communication terminal of the other party. . The other party's communication terminal can display an image on a display and output audio from a speaker to make a video call.

A technique for recording images during a conference has been devised (see, for example, Japanese Patent Application Laid-Open No. 2002-200011). Patent Literature 1 discloses a system that efficiently reproduces a conference by temporarily recording an image before processing so that it can be edited later.

However, the conventional technology has a problem that the second image is displayed superimposed on the first image, so even if the user later reproduces the recorded image, the first image cannot be displayed. be. In other words, the communication terminal displays not only the camera images of multiple sites (second image) but also the material (first image) on the display. display the material as large as possible. A picture-in-picture display method is used so that images from multiple sites are superimposed on materials in a reduced state. By doing so, the following usage becomes possible.
・All screens can be viewed at the same time.
・Important screens and materials (shared materials, base images during speech, etc.) can be enlarged.

However, even if the recording system records the materials displayed on the display and the camera images of each site, part of the first image (the image displayed in large size) is hidden, and the user cannot see the first image. Unable to display areas of concern.

SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide a video recording system capable of displaying a first image during playback even when a second image is superimposed on a first image.

In view of the above problems, the present invention provides a communication unit that receives first image data from another base, and second image data received from another base or prepared at one's own base for the first image. an image processing unit for moving the position of the first image data or the second image data within the frame of the moving image embedded with the moving image file in which the second image data is embedded in the first image data an image compositing unit that constructs the first image data that does not include the second image data from the moving image file; and the first image data constructed by the image constructing unit is displayed. and a display control unit for recording.

It is possible to provide a recording system capable of displaying the first image during playback even if the second image is recorded while being superimposed on the first image.

1 is an example of a schematic diagram of a system configuration of a communication system; FIG. 1 is an example of a configuration diagram of a video recording system at one site; FIG. 1 is an example of a hardware configuration diagram of a communication terminal; FIG. 1 is an example of a hardware configuration diagram of a recording/playback device; FIG. FIG. 2 is an example of a functional block diagram illustrating functions of a communication terminal and a recording/reproducing device divided into blocks; It is a figure explaining the information memorize|stored in a synthetic image memory|storage part. It is a figure explaining the information memorize|stored in a camera image memory|storage part. It is a figure explaining the information memorize|stored in a document image memory|storage part. FIG. 10 is a diagram showing an example of a conventional composite image created by a communication terminal; FIG. 10 is a diagram illustrating an outline of image processing for moving the position of a camera image in a synthesized image; FIG. 10 is an example of a flowchart illustrating recording processing of a composite image; It is the figure which represented the movement of the camera image of a synthetic image by the screen display image. FIG. 12 is an example of a flowchart for explaining the process of step S7 in FIG. 11; FIG. 4 is a diagram showing an example of metadata attached to a frame of a synthesized image; FIG. FIG. 4 is a diagram for explaining a method of reproducing a material image in the recording/reproducing device; FIG. 10 is an example of a flowchart for explaining a process of saving a camera image (moving image file) and a material image by a recording/reproducing device; FIG. 10 is a diagram for explaining an outline of image processing for creating a composite image while sliding a document image upward; It is a figure explaining up-and-down slide of a material image. FIG. 12 is a flowchart for explaining the process of step S7 in FIG. 11 in the process of sliding the material image upward; FIG. 10 is an example of a flowchart illustrating processing for constructing a material image by a second material image constructing unit in processing for vertically sliding the material image. FIG. 10 is a diagram showing three types of images that can be used when playing back a composite image; FIG. 10 is a diagram showing an example of image display using two displays;

DESCRIPTION OF THE PREFERRED EMBODIMENTS A video recording system and an image processing method performed by the video recording system will be described below as an example of embodiments of the present invention with reference to the drawings.

<Outline of recording system operation>
In the recording system of this embodiment, at least the following two images are transmitted and received between sites.
1. Camera image (movie) of the site, such as conference participants, captured by a camera
2. Document images to be shared at the meeting (generally static images)
The communication terminal at each base puts the images 1 and 2 on one screen (frames for one screen) while maintaining the visibility of the materials (makes them visible at the same time). In this case, the user has no choice but to use a layout similar to Picture In Picture (hereinafter referred to as embedding 1 in 2).

If the entire screen on which the camera image (an example of the second image data) and the material image (an example of the first image data) are laid out is recorded by the communication terminal, the person who could not participate in the meeting can play it back and review it. If
・By carefully reviewing the video of all time, it may be possible to grasp the blind spot area in the document image, but it takes time.
・Alternatively, we cannot deny the possibility that the area of the blind spot remains invisible.
There is an inconvenience.

<Outline of processing in this embodiment>
Therefore, in this embodiment, the following processing is performed so that the recording system can reproduce the area that was not visible (hidden) during the conference at an arbitrary timing during reproduction.

◆ When recording (processing of communication terminal)
The communication terminal executes (at least one of) the following processes during recording.
A. When the camera image is embedded in the material image, the display position of the frontmost image (usually the camera image) is slowly moved continuously.
B. When the backmost image (usually a material image) is changed to a new page or the material itself is changed, the material image is slid vertically or horizontally to switch to the next page.

◆ During playback (processing by recording/playback device)
C (corresponding to A). Since the display position of the camera image in the foreground moves, the recording/reproducing device can also obtain the area hidden behind the document image. By temporarily saving it as a separate file during playback, the recording and playback device can display the entire document image and move the camera image to any position in the document image during playback. can. "Temporary during playback" means that the file is saved as one file like a general recording device, and the recording and playback device generates the material image as a separate file from the one file during playback.

In addition, since the material image is often the material used in the meeting, it can be treated as a still image. That is what makes such a playback method possible.
D (corresponding to B). To obtain information on a blind spot area by utilizing the fact that the material image is slid and switched. By temporarily saving it as a separate file during playback by the recording and playback device, the entire document image can be displayed during playback, and the camera image can be moved to any position in the document image. .

These processes C and D can be realized because, unlike during a meeting, a synthesized image in which a camera image is embedded in a material image can be recorded and displayed in real time. That is, since processing is performed as if editing a composite image that has already been recorded, the composite image can be reproduced and displayed.

As described above, the communication system according to the present embodiment prevents users who have not been able to participate in the meeting from reviewing the synthesized image from being played back without seeing the blind spot area. No need to chase.

<Terms>
A blind spot is an area where the second image data overlaps in the first image data. The blind spot area is an area invisible to the user.

Real-time means that if a certain process is currently executed, the result of the process can be obtained within a certain delay.

<Configuration example of recording system>
FIG. 1 is a schematic diagram of the system configuration of a communication system 100. As shown in FIG. In the communication system 100, communication terminals 10A to 10D located at bases A to D communicate via a communication network N such as the Internet. Sites A to D have recording systems 60 (communication terminals 10 and recording/playback devices 30). The recording/playback device 30 does not necessarily have to be arranged at all bases. The number (four) of communication terminals 10A to 10D in FIG. 1 is an example, and the number may be two or more (two or more bases). An arbitrary communication terminal among the communication terminals 10A to 10D is referred to as a "communication terminal 10".

The communication terminals 10A to 10D transmit captured camera images and voices to the communication management system 50. FIG. Also, the communication terminals 10A to 10D receive camera images from the communication management system 50. FIG. Note that it is not necessary for the self-site to receive the camera image of the self-site. Any communication terminal 10 among the communication terminals 10A to 10D transmits the document image to the communication management system 50. FIG. The material image is an image of the material of the conference that the communication terminals 10A to 10D have imported from the external PC or internally held. The communication terminals 10A-10D receive the material images from the communication management system 50. FIG. The document image may be originally registered in the communication management system 50 .

The communication terminals 10A to 10D of the bases A to D display the camera images captured by the communication terminals 10A to 10D and the document images. The same applies to voice data. It is assumed that the camera images transmitted by the communication terminals 10A to 10D are moving images, but they may be still images. Also, the communication terminals 10A to 10D may simply receive camera images and voices without transmitting them.

Each of the communication terminals 10A to 10D may be a dedicated teleconference terminal or a PC (Personal Computer) running an application. That is, the PC is normally used as a general-purpose information processing device, and operates as the communication terminal 10 when executing a teleconference application.

Further, the communication terminal 10 as a general-purpose information processing device may be a smart phone, a tablet terminal, a PDA (Personal Digital Assistant), a mobile phone, a projector, an electronic blackboard, a car navigation system, etc., in addition to the PC.

A communication management system 50 is also connected to the communication network N. FIG. The communication management system 50 manages and controls communications of the communication terminals 10A-10D. The communication management system 50 may be constructed by a single computer, or may be constructed by a plurality of computers in which each part (function, means, or storage part) is divided and arbitrarily assigned.

The communication management system 50 manages, for example, the communication terminals 10 participating in the same conference by associating them with session IDs. Based on the session ID, the communication management system 50 transmits images (material images, camera images) and voices from each site to other sites participating in the same conference. By doing so, teleconferencing becomes possible between other bases.

The communication management system 50 is realized by one or more information processing devices. The communication management system 50 may be realized by cloud computing, or may be realized by a single information processing device. Cloud computing refers to a form in which resources on a network are used without being aware of specific hardware resources. The communication management system 50 may exist on the Internet or may exist on-premises.

FIG. 2 shows a configuration diagram of a recording system 60 at one site. Recording system 60 has communication terminal 10 and recording/playback device 30 . A display 20 may be connected to the recording/playback device 30 . It is also possible to connect a display 20 to the communication terminal 10 .

A camera and a microphone may be externally connected to the communication terminal 10 . The camera captures a range within the angle of view of its own site to generate a camera image, and the microphone collects the sound of its own site to generate audio data. The communication terminal 10 displays the camera image on a built-in or external display 20 and transmits the image to other bases. The communication terminal 10 transmits voice data to another base.

The communication terminal 10 embeds a camera image of its own site and a camera image of another site into a material image prepared by its own site or another site to generate a composite image. The communication terminal 10 transmits the synthesized image and audio data to the recording/playback device 30 . The recording/reproducing device 30 records synthesized images and audio data generated during a conference to create a moving image file.

The recording/reproducing device 30 is pressed by each site or department, and is used mainly for the digest of the conference for users who did not participate in the conference to grasp the content of the conference.

The recording/reproducing device 30 reproduces the moving image file of the composite image after the conference ends according to the user's operation. The reproduced video is displayed on the display 20 . Alternatively, the recording/reproducing device 30 may serve as a Web server and transmit the reproduced composite image moving image file to the user terminal. Also, the recording/reproducing device 30 may upload the moving image file to the cloud. The recording/playback device 30 may be on the cloud.

In FIG. 2, the communication terminal 10 and the recording/playback device 30 are separate units, but the communication terminal 10 may have the function of the recording/playback device 30.

<Hardware configuration example>
Hardware configurations of the communication terminal 10 and the recording/playback device 30 included in the recording system 60 according to the present embodiment will be described with reference to FIGS. 3 and 4. FIG.

<<communication terminal>>
FIG. 3 is a diagram showing an example hardware configuration of the communication terminal 10. As shown in FIG. The communication terminal 10 includes a camera module 301, a video processing unit 303, a video CODEC unit 314, a video output processing unit 310, a microphone array 304, an audio output unit 307, an audio processing unit 308, a network processing unit 313, an overall processing unit 315, an operation A unit 316 , a RAM 306 , a recorder I/F unit 317 , a video characteristic analysis unit 302 , a RAM 305 and a Capture processing unit 312 are provided. Note that the camera module 301 may be configured to connect an external general-purpose camera.

The camera module 301 is an example of an "imaging device". The camera module 301 captures video of the conference scene. The camera module 301 has a lens 301a, an imaging unit 301b (image sensor), and a DSP 301c. The imaging unit 301b generates video data (RAW data) by converting the video condensed through the lens 301a into an electrical signal. The DSP 301c generates video data (YUV data) by performing known camera video processing such as Bayer conversion and 3A control on the video data (RAW data) output from the imaging unit 301b.

The video processing unit 303 performs various video processing such as cropping processing and scaling processing 303a on the video data (YUV data) output from the camera module 301 according to the purpose. For example, the video processing unit 303 acquires face detection information from the video characteristic analysis unit 302 and beamforming information from the audio processing unit 308, and generates a close-up video of the speaker. The generated video is transferred to the video output processing section 310 . Note that the video processing unit 303 uses the RAM 306 as a buffer when performing various video processing.

The video CODEC unit 314 encodes and decodes video data (video stream data) transmitted/received to/from another communication terminal 10 . For example, the video CODEC unit 314 encodes the video data output from the video processing unit 303 by the video endocer 314a, and transmits the encoded video data to the other communication terminal 10 via the network processing unit 313. . Alternatively, the video CODEC unit 314 encodes the video data layout-processed by the video output processing unit 310 and transmits the encoded video data to the other communication terminal 10 via the network processing unit 313 .

Further, for example, the video CODEC unit 314 acquires video data transmitted from another communication terminal 10 (video data encoded by the other communication terminal 10) via the network processing unit 313, and , to decode the video data. Video CODEC section 314 then outputs the decoded video data to video output processing section 310 . The video CODEC unit 314 is configured by a CODEC circuit or software using compression standards such as H.264/265, for example.

The image output processing unit 310 displays an image based on the image data on the display included in the touch panel unit 309 . The display may be an external general monitor.

For example, the video output processing unit 310 causes the display included in the touch panel unit 309 to display a video based on the video data decoded by the video CODEC unit 314 (that is, a video of another site). There are multiple cases of videos from other bases.

Also, for example, the image output processing unit 310 causes the display included in the touch panel unit 309 to display an image based on the image data output from the camera module 301 (that is, the image of the home base). The video from the camera module 301 may be a video in which the speaker has been subjected to close-up processing by the video processing unit 303 .

Also, for example, the image output processing unit 310 outputs an image based on the image data output from the capture processing unit 312 (that is, an image such as a material screen displayed on the external PC 311) to the display provided in the touch panel unit 309. display.

In this way, the video output processing unit 310 displays a wide variety of images, and since it is necessary to fit these video images related to the video conference into one frame, the video output processing unit 310 performs layout processing. In some cases, the displayed images are simply displayed side by side, or in other cases, the shared materials are enlarged and the images of the participants at each base are displayed in Picture In Picture format. The video output processing unit 310 changes the layout at any time according to the characteristics of the video being displayed and the layout at that time. Layout change (movement of camera images, etc.) in this embodiment is executed by this module.

The microphone array 304 has a microphone array 304a and an A/D converter 304b. The microphone array 304a collects voices of participants in the video conference and outputs voice signals (analog signals). The A/D converter 304 b converts the audio signal (analog signal) of the audio output from the microphone array 304 a into a digital signal and outputs the converted audio signal (digital signal) to the audio processing unit 308 .

The audio output unit 307 has a D/A converter 307B and a speaker 307a. The D/A converter 307B converts an audio signal (digital signal) transmitted from another communication terminal 10 into an analog signal. The speaker 307a is supplied with the audio signal (analog signal) converted by the D/A converter 307B, and outputs the voice of the participant of the video conference collected at the other site.

The audio processing unit 308 has a DSP 308a, an audio CODEC function 308b, a noise canceling (NR/EC) function 308c, an audio discrimination function 308d, and a beamforming function 308e. Predetermined audio processing (for example, audio CODEC processing, noise cancellation (NR/EC), audio discrimination, beam forming, etc.) is performed on audio data constituting data. Then, the audio processing unit 308 outputs the audio data after the audio processing to the audio output unit 307 . At the same time, the audio processing unit 308 performs echo cancellation (EC) processing on the audio data input to the microphone array 304 while grasping the audio data to be output to the audio output unit 307 . Then, the audio processing unit 308 outputs the audio data after the audio processing to the network processing unit 313 . Also, the sound processing unit 308 specifies the direction of the sound by the beam forming function, and the image processing unit 303 generates a close-up image of the speaker based on the information.

The network processing unit 313 has a NIC 313a that transmits encoded video data output from the video CODEC unit (encoder) 314 to another communication terminal 10 as a transmission destination via the network. The network processing unit 313 also receives encoded video data transmitted from another communication terminal 10 via the network. The network processing unit 313 then outputs the video data to the video CODEC unit (decoder) 314 . The network processing unit 313 also has a network bandwidth monitoring function (network state detection unit 313b) for determining encoding parameters (QP value, etc.). In addition, the network processing unit 313 has a function (partner station function determination unit 313c) that acquires information about the function and performance of the partner station in order to optimize the settings of the encoding parameters (QP value, etc.) and transmission mode. have.

The overall processing unit 315 performs overall control of the communication terminal 10 . The overall processing unit 315 includes a CPU 315a, a ROM 315b, an SSD 315c, a RAM 315d, and the like. For example, the overall processing unit 315 performs mode setting, status management, etc. of each module and each block according to an operator's instruction. The overall processing unit 315 also has a function of arbitrating the right to use the system memory (RAM) and the right to access the system bus.

Also, the overall processing unit 315 sets the imaging mode of the camera module 301 . The setting of the imaging mode of the camera module 301 can include automatic setting items (for example, photometric conditions) that are automatically set according to the environment, and manual setting items that are manually set by the operator's operation input. .

The overall processing unit 315 also makes settings related to layout processing performed by the video output processing unit 310 . The overall processing unit 315 selects and sets a predetermined display format, such as Picture In Picture display for preferentially displaying shared materials, or displaying a specific site in a large size.

These settings are made by the operator operating the operation unit 316 and stored in the memory (RAM) of the communication terminal 10 . These settings are then used by the video processing unit 303 .

The operation unit 316 includes various input devices (eg, touch panel, operation buttons, remote control, etc.). The operation unit 316 receives various inputs (for example, various settings, calling of conference participants, etc.) by operating various input devices by the operator.

The recording device I/F unit 317 combines the audio data output from the audio processing unit 308 and the video data generated by the video output processing unit 310 to form recording data, and records the synthesized data. It has an I/F function for outputting to the playback device 30 .

The video characteristic analysis unit 302 has a detection unit 302a and a motion determination unit 302b. The detection unit 302 a detects an area in which a person's face exists from the frame images forming the video data output from the camera module 301 . The motion determination unit 302b detects an area in which a person is moving from frame images forming video data output from the camera module 301 . The image characteristic analysis unit 302 outputs the detection result of each area to the image processing unit 303 . Note that the video characteristic analysis unit 302 uses the RAM 305 as a buffer when detecting each area.

The capture processing unit 312 captures the video input from the external PC 311 and transfers it to the video output processing unit 310 . The capture processing unit 312 has a page break (update) detection function as a function related to this embodiment.

The capture processing unit 312 detects whether or not a page break (update) has occurred in the material (screen) sharing video transferred from the external PC 311 .

The detection method is not particularly limited, but since it is video data of a PC screen and there is no element of noise, a simple verify check or sum check between frames may be used. In order to reduce the amount of calculation, the capture processing unit 312 may perform the above processing after lowering the resolution of the captured image.

The page break (update) of the screen here includes the fact that the external PC 311 is connected after the conference has started and video transfer for material (screen) sharing has started from the external PC 311 as a page break trigger.

- When the capture processing unit 312 detects a page break (update), it notifies the video output processing unit 310 of that fact.

- The video transferred from the external PC 311 is treated here as an image for material (screen) sharing, so it is not treated as a video like the camera module video. Therefore, an upper limit is also set for the frame rate (that is, a certain amount of page break (update) detection interval is secured).

- If the user wants to transfer the moving image being played back on the external PC 311 as a normal moving image without limiting the frame rate, the user can disable the input image from the camera module. The user inputs such settings into the Capture control when specifying the operating mode of the system.

<<Recording and playback device>>
FIG. 4 is a hardware configuration diagram of the recording/playback device 30. As shown in FIG. As shown in FIG. 4, the recording/playback device 30 includes a CPU 501, a RAM 502, an operation unit 503, a ROM 504, an input I/F 505, a CODEC 506, a recording/playback circuit 507, an encryption circuit 508, an output I/F 510, and It has NIC511.

The CPU 501 controls each block of the recording/reproducing device 30 according to a predetermined program stored in the ROM 504 in order to implement the recording function described in this embodiment.

The RAM 502 is used as a work area for the CPU 501 and also as a storage area for each processing program stored in the ROM 504 . The RAM 502 is used as a temporary storage destination for image data and audio data transferred from the communication terminal 10 . The RAM 502 is also used as a work memory for the CODEC 506 and recording/playback circuit 507 .

The operation unit 503 is composed of hard keys, a remote control, or the like, and is a general user interface for starting the recording/reproducing apparatus 30, setting the mode, and the like.

The input I/F 505 is an interface used when inputting image data and audio data transferred from the communication terminal 10 . The I/F can be a dedicated one, or a general HDMI (registered trademark) or DisplayPort.

A CODEC 506 is configured by a CODEC circuit such as H.264/265 or software in order to encode/decode a frame of image data (video stream data) input from the input I/F 505 . The data encoded by the encoding process is encrypted by the encryption circuit 508 and then stored in the storage device 509 (HDD, SSD, SD memory card, etc.). The data stored in the storage device 509 is the composite image moving image file itself.

A recording/playback circuit 507 performs a series of image processing described in this embodiment in order to facilitate reuse of recorded data.

The output I/F 510 outputs the image data transferred from the communication terminal 10 to the display. Alternatively, the output I/F outputs a synthesized image reproduced from the synthesized image data to the display. I/F standards include HDMI (registered trademark) and DisplayPort (registered trademark). Audio data is also included in the image recording data.

The NIC 511 can be connected to the Internet via a network such as a LAN (Local Area Network), and transfers composite image data to an external server or NAS.

<About functions>
FIG. 5 is an example of a functional block diagram for explaining the functions of the communication terminal 10 and the recording/playback device 30 by dividing them into blocks.

<<communication terminal>>
The communication terminal 10 has a camera image acquisition section 11 , a voice acquisition section 12 , a communication section 13 , a composite image transmission section 14 , a movement section 15 , a page switching section 16 , an image synthesis section 17 and an operation reception section 18 . These functions of the communication terminal 10 are realized by the hardware circuit of the communication terminal 10 shown in FIG. 3, but may be realized by the CPU executing a program. Also, the functions in FIG. 5 merely show the main functions that the communication terminal 10 has, and the communication terminal 10 may have functions other than those shown.

The camera image acquisition unit 11 acquires the camera image captured by the camera module 301 from the camera module 301 in real time. The voice acquisition unit 12 converts the voice collected by the microphone array 304 into PCM to generate voice data.

The communication unit 13 transmits camera images to the communication terminals 10 at other bases via the communication management system, and receives camera images from the communication terminals 10 at other bases via the communication management system. The communication unit 13 transmits voice data to the communication terminals 10 at other sites via the communication management system 50 and receives voice data from the communication terminals 10 at other sites via the communication management system 50 . The communication unit 13 receives the document image via the communication management system 50 when the communication terminals 10 at other bases share the document image. When the communication terminal 10 of its own site shares the document image, the communication unit 13 transmits the document image to the communication terminal 10 of another site via the communication management system 50 .

Regarding the image processing of method A, the moving unit 15 continuously moves the camera image slowly with respect to the document image.

The image synthesizing unit 17 arranges the camera image of the own base, the camera images of the other bases, and the document image to generate a synthesized image of one frame according to the user's operation or automatically. A frame is an individual still image in a moving image. The user can reduce these camera images and material images and arrange them in tiles, or embed the camera images in the material images (or vice versa). In this embodiment, a case where a camera image is embedded in a document image and arranged will be described.

In addition, the image synthesizing unit 17 attaches the position information/size of the camera image in the synthesized image and the identification information of the base to the frame as metadata or the like.

Regarding the image processing of B above, the page switching unit 16 detects switching of the document image and slides the document image vertically or horizontally to switch from the document image before switching to the document image after switching. In each frame of the composite image, the material image in the middle of switching is recorded while sliding. Supplementally, the switching of the material image itself is performed by the user's operation. The page switching unit 16 detects the switching of the material image by the user's operation, and prepares the material image after switching for the processing of B (the material image before switching has already been obtained). The page switching unit 16 switches the prepared document image after switching and the document image before switching while sliding.

Note that the moving section 15 and the page switching section 16 function as an image processing section 21 . Either one of the moving unit 15 and the page switching unit 16 may operate, or both may operate.

The image synthesizing unit 17 arranges the camera image of the own base and the camera images of the other bases in the document image generated by the page switching unit 16 and recorded while sliding, thereby generating a synthesized image of one frame.

The composite image transmission unit 14 repeatedly transmits the composite image to the recording/playback device 30 like a moving image. In the case of moving images, for example, 30 or more frames are transmitted one after another per second. In addition to transmitting the composite image in real time, the composite image transmission unit 14 may accumulate a certain amount of the composite image (when the recording/playback device 30 is not outputting the display during the meeting) and then transmit the composite image.

The operation accepting unit 18 accepts a user's operation on the communication terminal 10 .

<<Recording and playback device>>
The recording/playback device 30 includes a receiving unit 31, a moving image file creating unit 32, a first document image constructing unit 33, an operation accepting unit 34, a switching detecting unit 35, a second document image constructing unit 36, a display control unit 37, and a storage unit. It has a portion 49 . These functions of the recording/playback device 30 are implemented by the hardware shown in FIG. 4, but may be implemented by the CPU executing a program.

The receiving unit 31 receives the composite image from the communication terminal 10 and stores it in the composite image storage unit 41 . In this manner, the composite image storage unit 41 stores a composite image in which the camera image is embedded in the document image.

The moving image file creating unit 32 captures the camera image from the composite image based on the position information and size of the camera image, and creates a moving image file (an example of the second moving image file) composed of the camera image. Extracting a camera image from a synthesized image is sometimes called trimming. The moving image file creating unit 32 stores the camera images of the same location in time series in the camera image storage unit 42 based on the identification information of the location. Since the camera image can be obtained from each frame of the composite image, the same fps (frames per second) frame as the composite image is saved for each camera image (for each site).

As the processing of C above, the first document image constructing unit 33 performs an OR operation on synthesized images captured at different times (a synthesized image obtained by cutting out a camera image is referred to as an intermediate image) to construct a document image. The first document image constructing unit 33 stores the constructed document image in the document image storage unit 43 . This document image is built up every time it takes for the camera image to move until there is no blind spot.

As the processing of D above, the switching detection unit 35 detects that the document image itself or the page of the document image (hereinafter referred to as page switching without distinction) has been switched. The switching detection unit 35 compares the areas other than the camera image of the composite image for each frame having different imaging times. If the difference is equal to or greater than a certain value, the switching detection unit 35 detects that the page of the document image has been switched. Alternatively, the switching detection unit 35 detects page switching of the document image based on a page switching operation signal transmitted from the communication terminal 10 .

As the processing of D above, the second document image constructing unit 36 constructs a document image by combining the document images of the portions where the camera images do not overlap when it is detected that the page has been switched. The second document image constructing unit 36 stores the constructed document image in the document image storage unit 43 . This material image is constructed each time the page is switched.

Either one of the first document image constructing section 33 and the second document image constructing section 36 may perform the processing, or both of them may perform the processing. If both process, the same document image may be created, but the same image may be deleted.

The operation accepting unit 34 accepts playback operations and recording operations for the recording/playback device 30 . The display control unit 37 displays the synthesized image, camera image, or document image on the display 20 .

FIG. 6 is a diagram for explaining information stored in the composite image storage unit 41. As shown in FIG. In the composite image storage unit 41, one moving image file is stored from the start to the end of recording. In FIG. 6, items such as recording ID, imaging start time, imaging end time, number of bases, size, recording time, and file name are managed for each moving image file.
- A recording ID is identification information for identifying a moving image file, and is information for associating a camera image or a document image with a composite image.
- The imaging start time is the recording start time.
- The imaging end time is the recording end time.
- The number of bases is the number of bases participating in the conference, and is transmitted from the communication terminal 10 . Note that the number of sites includes the own site.
・The size is the capacity of the video file.
- The recording time is the elapsed time from the start time to the end time.
・The file name is the file name of the composite image that is saved along with the file path. The video file format is not limited.

FIG. 7 is a diagram for explaining information stored in the camera image storage unit 42. As shown in FIG. In the camera image storage unit 42, camera images for each site are stored as one moving image file. In FIG. 7, the items of recording ID, imaging start time, imaging end time, site ID, size, and recording time are managed for each moving image file. The recording ID, imaging start time, imaging end time, size, and recording time may be the same as in FIG.
- The base ID is identification information for identifying a base. The recording/reproducing device 30 may acquire the site name from the communication terminal 10, or may assign a unique number.

Although the file names are omitted in FIG. 7, the camera images for each location are saved as moving image files. Also, the camera images of each site may be saved in one moving image file.

FIG. 8 is a diagram for explaining information stored in the document image storage unit 43. As shown in FIG. FIG. 8(a) shows the information of the document image constructed by the first document image construction unit 33, and FIG. 8(b) shows the information of the material image constructed by the second material image construction unit 36. FIG.

In the document image storage unit 43 shown in FIG. 8A, one or more document images constructed from moving image files of composite images are stored. In FIG. 8A, the items of recording ID, construction start time, construction end time, material image ID, size, and file name are managed for each material image.
The construction start time is the time when the construction of the document image is started based on the recording start time. The start of construction may be the time of state A, which will be described later.
• The construction end time is the time when the construction of the material image was completed with reference to the recording start time. The construction may end at the time of state C, which will be described later.
- The document image ID is identification information for identifying the document image.
・The size is the capacity of the document image.
・The file name is the file name of the document image. Although the document image is a still image in FIG. 8A, it may be saved as a moving image.

In the document image storage unit 43 shown in FIG. 8B, items such as recording ID, page switching time, document image ID, size, and file name are managed for each document image. The items of recording ID, material image ID, size, and file name may be the same as those in FIG. 8(a).
The page switching time is the time at which the page switching was detected and recorded with reference to the recording start time.

<Conventional synthetic image method>
FIG. 9 is an example of a conventional composite image created by the communication terminal 10. As shown in FIG. The composite image in FIG. 9 has the following configuration.
- A material image 201 under discussion is displayed on the entire screen. Since it contains detailed text, it is desirable to display it as large as possible.
A panorama image 202 (camera image) in which all participants at a site are captured is shown at the bottom left of the screen.
The bottom center of the screen is an individual image 203 (camera image) for each individual who participates remotely because of working from home or the like. One of them is the camera image captured by the camera at the base.
- The bottom right of the screen is the speaker camera image 204 in which the person who is speaking is displayed in a slightly larger size.

If the recording/reproducing apparatus 30 records the synthesized image of FIG. 9 as it is, the following problems arise when a person who could not participate in the conference reproduces the image.
・Blind spots in materials cannot be avoided.
・Blind spots may be eliminated if the material image is displayed in a small size.
・There may be times when the blind spot area is visible, but if the user does not know the times when the blind spot is visible, the user may need to watch the entire video during playback. In that case, there is a method that is difficult to watch the reproduced video in order to save time, but in the end, there may be a case where the user cannot see the video to the end.

Therefore, the synthesized image is a recorded video that is difficult to use for those who could not participate in the conference.

<Overview of image processing according to the present embodiment>
Two types of image processing for a composite image are described below.

A. Movement of Position of Camera Image First, FIG. 10 is a diagram for explaining an outline of image processing for moving the position of a camera image in a synthesized image.

The moving unit 15 of the communication terminal 10 slowly and subtly changes the positions of the camera images of each base in the order of FIG. 10(a)→FIG. 10(b)→FIG. 10A to 10C, it can be seen that the panorama image 202, the individual image 203 of the home remote participant, and the video of the speaker are moving. The moving unit 15 moves so as not to leave any unseen regions by changing the position of the camera image. For example, the image synthesizing unit 17 grasps the "window size (size) of the camera image" and the "display position coordinates (position information)", and slowly moves the position (larger than the size) so as not to leave any blind spots. changing.

In the recording/reproducing device 30, the composite image is recorded after being compression-encoded. At the time of playback (or between recording and playback), the first document image construction unit 33 of the recording/playback device 30 uses the change in the display position of the camera image as the process C to eliminate blind spots. Separately create a material image. Since the meeting has already ended (because it doesn't have to be in real time), such an operation is possible.

Also, during playback, the recording/playback device 30 can display only material images without blind spots. If there are two displays, the document image and the camera image can be displayed simultaneously on separate displays.

With the above operations, it is expected that the time required to grasp the content of the conference during playback can be greatly reduced.

<Processing procedure>
FIG. 11 is a flowchart for explaining recording processing of a composite image.

A user of the communication terminal 10 activates the communication terminal 10 or an application. The user performs initial settings related to the recording system 60 at startup (S1). The initial settings are, for example, designation of a connection destination site, layout designation of a conference screen, and the like.

Next, the communication terminal 10 initializes input/output devices such as a camera, a microphone, and a speaker, sets operation modes, etc., and starts each device (S2). For example, the communication terminal 10 sets the camera imaging mode such as photometric conditions in accordance with the conference environment.

When the main body is ready in steps S1 and S2, the communication section 13 requests the communication management system 50 to start communication and starts the conference (S3). The communication unit 13 may start communication upon receiving a communication request from the partner station. When the communication terminal 10 records the composite image, the recording/playback device 30 is also activated at this time.

If the composite image is in the recording state, the process moves to step S7. Otherwise, the process proceeds to step S5.

If the recording/playback device is in the READY state, the process proceeds to step S6. Otherwise, the process proceeds to step S8.

The composite image transmission unit 14 of the communication terminal 10 transmits the composite image to the recording/reproducing device 30, so that the recording/reproducing device 30 starts recording processing (S6).

The composite image transmission unit 14 executes or continues the recording process (S7). When the conference ends, the communication terminal 10 shifts to a standby state (S8). The process returns to step S4 and repeats the processes from S4 to S7 until the conference ends.

<<Move the position of the camera image>>
FIG. 12 is a diagram showing a screen display image of movement of a camera image of a composite image. In other words, if the moving unit 15 moves the camera image as shown in FIG. 12, blind spots in the document image can be eliminated.
1. During the transition from state A to state B, the panorama image 202 and the speaker camera image 204 move as shown in the drawing (upward movement).
2. During the transition from state B to state C, the individual image 203 and the speaker's camera image 204 move as shown in the figure (the individual image 203 moves leftward, and the speaker's camera image 204 moves upward).
3. During the transition from state C to state D, the individual image 203 and the speaker's camera image 204 move as shown in the figure (the individual image 203 moves rightward, and the speaker's camera image 204 moves downward). That is, each camera image tries to return to its original position.
4. During the transition from state D to state A, the panorama image 202 and the speaker camera image 204 move as shown in the drawing (moving downward, trying to return to the initial position).
This state transitions as follows.
State A→state B→state C→state D→state A→state B→state C→state D→... (repeat thereafter)

FIG. 13 is a flow chart for explaining the process of step S7 in FIG. In FIG. 13, the movement direction of the camera image is designated according to the current movement state. As described with reference to FIG. 12, the states A to C during movement of the camera image are any of the three states (states D to A can be omitted as states because they are on the return trip), so the camera image is as follows. be moved.

When the camera image is being gradually moved from state A to state B (Yes in S71), the moving unit 15 gradually moves the camera image from state A to state B (S72). If step S71 is No, the process proceeds to step S73.

When the camera image is being gradually moved from state B to state C (Yes in S73), the moving unit 15 gradually moves the camera image from state B to state C (S74). If step S73 is No, the process proceeds to step S75.

When the camera image is being gradually moved from state C to state D (Yes in S75), the moving unit 15 gradually moves the camera image from state C to state D (S76). If step S75 is No, the process proceeds to step S77. The moving unit 15 gradually moves the camera image from state D to state A (S77).

The mobile unit 15 of the communication terminal 10 is always performing the processing of steps S72, S74, S76, S77. For example, the time from state A to state C is set in advance, and may be a slow time such as 30 seconds or 1 minute (a time during which the user does not notice the movement of the camera image).

FIG. 14 is an example of metadata attached to frames of a synthesized image. A frame is attached with a frame ID, region ID, location information, size, and state.
- The frame ID is, for example, identification information of a frame. It is preferable to include the imaging time and the elapsed time from the start of imaging.
- The base ID is identification information for identifying a base.
- The position information is the coordinates of the camera image in the composite image. The position information is a value controlled by the moving unit 15 .
・The size is the width and height of the camera image. The size may be set by the user or fixed, but is known anyway.

Site IDs, location information and sizes are included for the number of camera images.
The states A to D are stored in the state when the arrangement of the camera image is the above A to D. It is a value controlled by the moving unit 15 .

<C. Construction of Material Image in Recording/Playback Device>
15A and 15B are diagrams for explaining a method of reproducing a material image in the recording/reproducing device 30. FIG. The first document image constructing unit 33 of the recording/reproducing device 30 utilizes the recorded composite image to construct a document image without blind spots.

The first document image constructing unit 33 acquires each frame of state A, state B, and state C from the image data of the composite image. Whether the frame is in state A, state B, or state C may be determined by the recording/reproducing device 30 based on the position of the camera image, or may be determined by the image synthesizing unit 17 from the metadata shown in FIG. good.

The first document image constructing unit 33 replaces the area where the camera image was displayed with black pixels (0 in 256 gradations) to create an intermediate image 250 (FIGS. 15(a) to 15(c)). The intermediate image 250 may be a synthesized image after trimming the camera image by the moving image file creating section 32 . The first document image constructing unit 33 may be replaced with white pixels. The positions and sizes of camera images in state A, state B, and state C can be obtained from metadata.

The first document image constructing unit 33 can construct a document image 251 without blind spots by synthesizing the intermediate images 250 of state A', state B', and state C' by OR operation for each pixel position. (d)). This OR operation is, for example, a process of discarding if the pixel value of the three intermediate images 250 is 0, and adopting the average (or any of an arbitrary image) if the pixel value is not 0.

Since there is a possibility that the document image is switched between states A to C (while the position of the camera image is changing), the first document image constructing unit 33 selects the area of the intermediate image 250 where there is no camera image. It is preferable to divide into blocks and compare the blocks of the three intermediate images 250 to detect switching. If the document image has been switched, the first document image constructing unit 33 does not construct the document image. Alternatively, if there are two document images that have not been switched, the document image may be constructed (in this case, the entire document image may not be constructed).

In the above description, three composite images are used for states A to C, but it is sufficient if there are composite images so as to eliminate blind spots, and two or four or more may be used.

<Creation of document images and video files by processing C>
FIG. 16 is a flowchart for explaining the process of saving the camera image (moving image file) and the material image by the recording/reproducing device 30. As shown in FIG. The process of FIG. 16 can be executed when the moving image file of the composite image is stored in the composite image storage unit 41. FIG. It may be started when the user starts an operation to reproduce the composite image on the recording/reproducing device 30 (ordinary data storage may be performed with the composite image as it is).

First, the moving image file creation unit 32 acquires a composite image from the composite image storage unit 41 (S11). The synthesized image to be acquired may be a moving image file specified by the user, or may be a moving image file of camera images or a moving image file in which material images are not constructed.

The moving image file creating unit 32 cuts out the camera image from the frame based on the position information and size of the camera image included in the metadata of each frame. The moving image file creation unit 32 stores these as moving image files in the camera image storage unit 42 for each site (S12).

Next, the first document image constructing unit 33 identifies frames with statuses A to C from the composite image moving image file (S13). Here, it is assumed that intermediate images have been created by the moving image file creation unit 32 .

Frames of states A to C appear in time series at substantially regular intervals. The first document image constructing unit 33 constructs a document image by OR-operating frames of states A to C as one set (S14).

The first document image construction unit 33 stores the document image in the document image storage unit 43 in association with the recording ID, construction start time, construction end time, document image ID, size, and file name (S15). The construction start time is the state A frame imaging time, and the construction end time is the state C frame imaging time.

<B. Vertical slide of the material image>
Next, generation of a composite image and construction of a recorded material image by "B. Vertical slide of material image" will be described.

FIG. 17 is a diagram for explaining an outline of image processing for creating a composite image while sliding a material image upward.

The page switching unit 16 of the communication terminal 10 switches the material image to the next material image (next page) while moving the material image upward as shown in FIG. transition to The slide speed is set to a speed that allows the recording/reproducing device 30 to capture the material image. In FIG. 17, the camera image is below the document image, so the page switching unit 16 slides the document image upward. slide to the right. When the camera image is above the document image, the page switching unit 16 slides the document image downward. In this manner, the page switching unit 16 determines the slide direction in consideration of the layout of camera images.

17(a)→FIG. 17(b)→FIG. 17(c), the material image A before switching and the material image B after switching gradually slide upward.

A detailed description will be given with reference to FIG. FIG. 18 is a diagram for explaining vertical sliding of the document image. First, the page switching unit 16 detects switching of the document image as shown in FIG. 18(a). For example, the page switching unit 16 can compare document images transmitted from other bases at regular intervals and detect switching of the document images. The page switching unit 16 may detect an operation signal indicating that the page has been switched. The page switching unit 16 prepares the material image B after switching.

When the switching of the document image is detected, the page switching unit 16 connects the two document images A and B in the vertical direction, and as shown in FIG. 18(b)→FIG. 18(c)→FIG. , a portion within one frame 220 is cut out as a material image. The page switching unit 16 gradually moves the range to be cut from the upper side of the two document images A and B to the lower side. For example, in FIG. 18B, the entire document image A before switching is arranged in one frame 220, and in FIG. is arranged in one frame 220, and the entire document image B after switching is arranged in one frame 220 in FIG. 18(d). The image synthesizing unit 17 creates a synthetic image by arranging the camera images in such a frame.

Then, in the recording/reproducing device 30, the composite image described with reference to FIGS. 17 and 18 is recorded after being compression-encoded. This composite image becomes a moving image in which the material images are gradually switched while sliding up and down. The camera image is also embedded in the document image. When reproducing the material image, the second material image constructing unit 36 of the recording/reproducing device 30 uses the fact that the material image is displayed while sliding as processing D to separately create a material image without blind spots. .

That is, the second document image constructing unit 36 constructs a document image by combining the document images of the portions where the camera images do not overlap. Assuming that the area where the camera image exists is L from the lower end, the area of L or more should be cut out. The second document image constructing unit 36 cuts out, for example, the upper half 221 (part of the frame) of the screen (frame) in FIG. part). The second document image constructing unit 36 combines the two clipped document images to construct a document image. If the switching time from FIG. 18(b) to FIG. 18(d) is assumed to be T seconds, the second document image constructing unit 36 immediately before the detection of the switching and after T/2 seconds after the detection of the switching, the upper frame of each frame. The halves 221 and 222 should be cut off.

The image processing in FIGS. 17 and 18 is performed when there are many conference participants (because of the large number of participating sites), when it is difficult to move the camera image of A, or when page breaks in the material occur at fairly short intervals. Suitable for

<Processing procedure>
FIG. 19 is a flow chart for explaining the process of step S7 in FIG. 11 in the process of sliding the document image upward.

The page switching unit 16 determines whether or not the document image (page) has been switched based on the document image transmitted from another site and the document image being displayed (S21). The page switching unit 16 can detect page switching by a user's operation when the communication terminal 10 at its own site displays a material image.

Also, the page switching unit 16 determines the slide direction based on the position of the camera image in the document image (S22). Here, it is assumed that the page switching unit 16 has determined to slide upward.

Next, the page switching unit 16 connects the two document images vertically with the document image before switching on the top and the document image after switching on the bottom (S23).

The page switching unit 16 divides the height of the upper material image by the time T to determine the slide speed (S24). The time T is preset. It is preferable that the time T be as fast as possible (for example, 10 seconds) as long as the capture process can be completed in time. This is to display the material image so that the user can keep up with the discussion during the meeting.

The page switching unit 16 determines the range to be included in the frame based on the slide speed for each frame of the synthesized image, with reference to the upper end of the upper material image (S25).

The page switching unit 16 repeats the process of step S24 until the upper material image does not fit in the frame (S26).

<D. Construction of Material Image in Recording/Playback Device>
Next, referring to FIG. 20, construction of a material image in the recording/playback device 30 will be described. FIG. 20 is a flowchart for explaining the process of constructing a material image by the second material image construction unit 36 in the process of vertically sliding the material image.

First, the second document image constructing unit 36 acquires the moving image file of the composite image from the composite image storage unit 41 (S31). The synthesized image to be acquired may be a moving image file specified by the user, or may be a moving image file of camera images or a moving image file in which material images are not constructed.

The switching detection unit 35 monitors the frame of the synthesized image and determines whether or not the material image has been switched (S32).

If the determination in step S32 is Yes, first, the second document image constructing unit 36 captures the upper half of the document image before switching (S33). Since the composite image has already been saved, the second material image constructing unit 36 can specify the frame immediately before the start of switching.

Next, the second document image constructing unit 36 captures the upper half of the document image after T/2 seconds have passed since the switching was detected (S34).

The second document image constructing unit 36 connects the upper half and the lower half of the document image to construct the document image (S35).

The second document image constructing unit associates the document image with the recording ID, page switching time, document image ID, size, and file name, and stores the document image in the document image storage unit 43 (S36).

The second document image constructing unit 36 repeats the processing of FIG. 20 until the end of the composite image moving image file (S37).

<Browsing of material images by the operation of a user who did not participate in the meeting>
FIG. 21 is a diagram showing three types of images that can be used when reproducing a synthesized image. FIG. 21(a) shows a composite image 260. FIG.

FIG. 21(b) is a file for a material image.
- FIG. 21(b) is a file of a material image 251 with no blind spots for materials, etc. shared during the meeting, created by the process of C or D. FIG.

FIG. 21(c) is a moving image file displaying only camera images 252 such as the panorama image 202, the individual image 203, and the speaker camera image 204. FIG.
・If it is not necessary to display the material at the same time as the material during playback, it is possible to enlarge the display, making it easier to understand the facial expressions of people.

FIG. 21(d) is a moving image file of a composite image 260 similar to FIG. 21(a).
• Fig. 21(d) is the original composite image, which can be displayed as a moving image.

An example of a use case will be explained. A user who does not participate in the conference connects a PC to the recording/reproducing device 30 and selects a moving image file for a composite image. The user can determine which composite image is for the desired conference from the imaging start time or the like. As shown in FIG. 21(d), the user browses the composite image from the beginning, but when he becomes concerned about the blind spot of the material, he double-clicks the material, for example. The operation reception unit 34 detects the request for materials. The display control unit 37 detects the display time of the composite image for which the material is requested, and acquires the material image corresponding to the display time from the material image storage unit 43 . For example, if a material is requested when 1 minute and 15 seconds have elapsed from the start of the composite image, a material image of 001-003.jpeg constructed from frames between 1 minute and 1 minute and 30 seconds is obtained. The display control unit 37 can display this document image (without blind spots) in a large size on the display.

Similarly, when the user is concerned about the facial expressions of the participants, he/she double-clicks an arbitrary camera image, for example. The operation reception unit 34 detects the request for the camera image. The display control unit 37 detects the display time of the composite image for which the camera image is requested, and acquires the camera image corresponding to the display time from the camera image storage unit 42 . For example, if a camera image is requested after 1 minute and 15 seconds have passed since the start of the composite image, the camera images (moving image files) of each site are played back after 1 minute and 15 seconds have passed.

Also, as shown in FIG. 22, if there are two displays 20A and 20B, the recording/reproducing device 30 can display the camera image 252 and the document image 251 at the same time. FIG. 22 shows a display example using two displays of a camera image 252 and a material image 251 created from a composite image 260. FIG. By separately preparing the camera image and the document image as described with reference to FIG. 21, the following reproduction method is possible.
- If the recording/playback device 30 has two output I/Fs, the images shown in FIGS. 21(b) and (c) can be simultaneously output to the two displays 20A and 20B, respectively.
- Alternatively, by using the multi-stream function of Dual Display and DisplayPort (which can connect displays), the images in FIGS. 21(b) and (c) can be output to the monitor at the same time.

<Main effects>
As described above, the recording system 60 of the present embodiment gradually moves the camera image, or slides the material image to switch pages, so that the recording/reproducing device 30 can reproduce the image. You can also get the area that was hidden behind. By storing it as a separate file, the recording/playback device 30 can display the entire document image or move the camera image to an arbitrary position of the document image during playback.

<Other application examples>
Although the best mode for carrying out the present invention has been described above using examples, the present invention is by no means limited to such examples, and various modifications can be made without departing from the scope of the present invention. and substitutions can be added.

For example, in this embodiment, an example in which a camera image is embedded in a document image has been described, but this embodiment can be applied regardless of what is shown in the image. For example, the material image may be a landscape image. The camera image may be an image provided by a video posting site. Alternatively, the document image may be the web page, and the camera image may be the advertisement.

Also, in the present embodiment, the recording/reproducing device 30 constructs the material image, but the server may construct the material image.

Further, the configuration examples in FIG. 5 and the like are divided according to main functions in order to facilitate understanding of processing by the communication terminal 10 and the recording/playback device 30 . The present invention is not limited by the division method or name of the unit of processing. The processing of the communication terminal 10 and the recording/playback device 30 can also be divided into more processing units according to the content of the processing. Also, one processing unit can be divided to include more processing.

Each function of the embodiments described above may be implemented by one or more processing circuits. Here, the "processing circuit" in this specification means a processor programmed by software to perform each function, such as a processor implemented by an electronic circuit, or a processor designed to perform each function described above. Devices such as ASIC (Application Specific Integrated Circuit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array) and conventional circuit modules are included.

10 communication terminal 30 recording/playback device 60 recording system 100 communication system

Japanese Patent Application Laid-Open No. 2009-18298

Claims (10)

a communication unit that receives the first image data from another base;
The position of the first image data or the second image data in the moving image frame in which the second image data received from another site or prepared at the own site is embedded in the first image data an image processing unit that moves the
an image synthesizing unit that generates a moving image file in which the second image data is embedded in the first image data;
an image construction unit that constructs the first image data that does not include the second image data from the moving image file;
a display control unit that displays the first image data constructed by the image construction unit;
A recording system with arranging the first image data in a moving image frame, and taking a preset time to make the second image data larger than the size of the second image data with respect to the first image data; Having a moving part for moving,
The image constructing unit acquires a plurality of the first image data in which the positions of the second image data are different from the moving image file, extracts the second image data from the first image data, and synthesizes the second image data. 2. The recording system according to claim 1, wherein the first image data that does not contain the second image data is constructed by doing so. The image constructing unit performs an OR operation on a plurality of the first image data obtained by removing the second image data from the first image data, thereby creating the first image that does not include the second image data. 3. The recording system according to claim 2, wherein the recording system constructs data. When it is detected that the first image data has been switched,
a page switching unit that takes a preset time to slide the first image data before switching arranged in a frame of a moving image to switch to the first image data after switching;
The image synthesizing unit generates a moving image file in which the second image data is embedded in the first image data in the middle of the slide,
When it is detected that the first image data has been switched from the moving image file,
The image constructing unit cuts out a portion of the first image data before switching and a portion of the first image data after sliding, and synthesizes them to form the second image data. 2. The video recording system according to claim 1, wherein said first image data that does not contain . When it is detected that the first image data has been switched from the moving image file,
The image constructing unit cuts out half of the first image data before switching and half of the first image data after half of the time, and synthesizes the second image data. 5. The video recording system according to claim 4, wherein said first image data that does not contain . the moving image file has position information and size of the second image data in the first image data;
6. The method according to any one of claims 1 to 5, further comprising a moving image file creation unit that creates a second moving image file of said second image data captured from said first image data based on said position information and size. The recording system according to any one of items 1 and 2. 7. The recording system according to any one of claims 1 to 6, wherein said first image data not including said second image data is displayed according to a user's operation. 7. The recording system according to claim 6, wherein said second moving image file is displayed according to user's operation. a communication unit that receives the first image data from another base;
The position of the first image data or the second image data in the moving image frame in which the second image data received from another site or prepared at the own site is embedded in the first image data an image processing unit that moves the
a receiving unit for receiving the moving image file from a communication terminal having an image synthesizing unit for generating a moving image file in which the second image data is embedded in the first image data;
an image construction unit that constructs the first image data that does not include the second image data from the moving image file received by the reception unit;
a display control unit that displays the first image data constructed by the image construction unit;
A recording/playback device having a step in which the communication unit receives the first image data from another base;
The image processing unit embeds the first image data or the second a step of moving the position of the image data of
an image synthesizing unit generating a moving image file in which the second image data is embedded in the first image data;
an image construction unit constructing the first image data that does not include the second image data from the moving image file;
a step in which a display control unit displays the first image data constructed by the image construction unit;
An image processing method comprising:

Images