JP2009171246A - Communication terminal device, and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To achieve smooth understanding in a conference by displaying material corresponding to a pointer graphic when proceeding with the conference while presenting the material, for example, in a video conference system; and further to more simplify and more improve efficiency for operation, control processing and the like relating to graphic display. <P>SOLUTION: Between near-end and far-end communication terminal device sides, an image signal inputted at the near-end side is commonly displayed out. A mark image is combined with images displayed at the near-end side and a far-end side thereafter, and displayed. For the mark image, a picture pattern thereof is selected from among the predetermined number of picture patterns by operation at the near-end communication terminal device side. Furthermore, in order to display the mark image at the far-end communication terminal device side, control information for pointing graphic image display is transmitted from the near-end communication terminal device to the far-end communication terminal device, and the far-end communication terminal device displays thereon the pointing graphic image on the basis of the control information. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a communication terminal device that is a component of a video conference system, for example, and an image processing method applicable to the communication terminal device.

In the video conference system, a terminal device compatible with the video conference system is provided for each place such as a conference hall separated from each other, and an imaging device and a display device are connected to the terminal device. Then, in one conference hall as one near end, for example, a captured image obtained by imaging a conference participant or the like in the conference hall with an imaging device is connected to another conference hall at the far end via the terminal device. To the terminal device. The far-end terminal device receives the captured image transmitted from the near-end side and displays it on a display connected to itself.
In addition, a microphone and a speaker are also connected to the terminal device, and a near-end speaker's voice signal obtained by collecting sound by the near-end microphone is transmitted to the far-end side via the terminal device. The far-end terminal device outputs the received audio signal from the speaker.
Thereby, depending on the video conference system, it is possible to proceed with the conference by having a conversation while displaying the state of the conference room of the communication partner on the display device.

In actual meetings, there may be situations where it is necessary to give explanations using materials. However, if you simply have a voice conversation while displaying the materials you have taken, you will communicate. May not be performed smoothly.
Therefore, in Patent Document 1, “the video from the superimposing means in one communication device is displayed on the monitors of both communication devices, and the position information and the mouse button operation information of one communication device are displayed. The drawing means describes that the drawing means generates a signal for displaying the pointer graphic on the monitor from these pieces of information and generates a display signal for dots, straight lines, and circles.

JP-A-8-163528

  In the present invention as well, in a video conference system, for example, when a conference is progressed by presenting materials etc., an object equivalent to a pointer graphic is displayed to facilitate communication in the conference. In addition, it is an object to further simplify and increase the efficiency of operations and control processes related to graphic display.

In view of the above-described problems, the present invention is configured as a communication terminal device as follows.
That is, display is performed from among image signal input means for inputting an image signal, communication means for performing communication between other communication terminal devices, and a predetermined number of instruction graphic images whose modes are determined. An operation means for selecting an instruction graphic image to be performed and an image signal input by the image signal input means are displayed and output as a main image or an image portion thereof, and an instruction graphic image is obtained by an operation on the operation means. In response to the selection, the display control means for controlling the selected graphic image for instruction to be synthesized and displayed in the main image, and the image signal input by the image signal input means, The image transmission means to be transmitted by the communication means as an image signal to be displayed and output as the main image or its image portion on the other communication device side, and the operation to the operation means. In response to the instruction graphic image being selected by the above, the selected instruction graphic image is synthesized and displayed in the main image displayed on the other terminal device side. Control information generating means for generating control information to be transmitted, and control information transmission control means for transmitting the control information by the communication means.

  Also, a communication means for executing communication with another communication terminal device, and an instruction graphic of any one of a predetermined number of instruction graphic images received by the communication means and defined in its form Control information holding means for holding control information transmitted from another communication terminal apparatus for displaying an image, and an image signal received by the communication means and transmitted from the other communication terminal apparatus, as a main image or Based on the control information held by the main image corresponding display control means and the control information holding means to be displayed and output as the image portion, and for the instruction to control the graphic image for indication to be synthesized and displayed in the main image And a graphic image display control means.

Depending on the above configuration, the image signal input on the near end side can be displayed and output in common between the near end communication terminal apparatus side and the far end (other) communication terminal apparatus side. In addition, the instruction graphic image can be combined with the images displayed on the near end side and the far end side and displayed.
Then, the instruction graphic image to be displayed is selected from a predetermined number of modes whose modes are determined by an operation on the near-end communication terminal device. That is, when the user tries to display the instruction graphic image, an operation of selecting an appropriate one from the instruction graphic images prepared in advance may be performed.
In addition, when displaying the instruction graphic image in the main image on the far-end communication terminal device side, the instruction graphic image is synthesized from the near-end communication terminal device to the far-end communication terminal device. Instead of transmitting the image signal of the main image, the control information for displaying the graphic image for instruction is transmitted, and the communication terminal device at the far end displays the graphic image for instruction based on this control information. Operate.

  In this way, the present invention enables display of an instruction graphic image for an image displayed in common on the near-end communication terminal device side and the far-end communication terminal device side. By using the graphic image by the user, communication in a video conference system or the like can be sufficiently communicated. In addition, the instruction graphic image is displayed in response to an operation of selecting from among those prepared in advance. As a result, the operation for displaying the instruction graphic image is greatly simplified, and the operability is improved accordingly. Further, since the mode of the instruction graphic image is determined in advance, the processing and algorithm for drawing and display can be simplified. Further, when the instruction graphic image is displayed on the far-end communication terminal device side, control information is transmitted, and control for displaying the instruction graphic image is executed on the far-end communication terminal device side. As a result, the communication process can be made light and efficient.

As the best mode for carrying out the present invention (hereinafter referred to as an embodiment), the present invention is applied to a video transmission / reception system in a video conference system (video conference system).
A video conference system installs a video conference terminal device, which is a communication terminal, for each conference hall at different locations. From this video conference terminal device, an image captured by an imaging device and audio collected by a microphone are transmitted to another video conference. The terminal device is configured to transmit the image and the sound transmitted from another video conference terminal device, and to output from the display device and the speaker, respectively. That is, the video conference system includes a video transmission / reception system that transmits / receives images to / from each other and an audio transmission / reception system that transmits / receives audio to / from each other. The present embodiment corresponds to a system in which a video conference terminal device provided for transmitting and receiving a captured image as the video transmission / reception system is a core.

FIG. 1 shows a system configuration example mainly including a video transmission / reception system in a video conference system.
In this case, two places A and B that are separated from each other are used as conference halls, and the video conference terminal apparatuses 1-1 and 1-2 are installed in these places A and B, respectively. These video conference terminal apparatuses 1-1 and 1-2 are connected by a communication line corresponding to a predetermined communication method so that mutual communication is possible.
In the place A, an imaging device 2-1, a pan head 3-1, and a display device 4-1 are provided together with the video conference terminal device 1-1.
The imaging device 2-1 includes a zoom mechanism for adjusting the angle of view, and is attached to the pan head 3-1. The camera platform 3-1 includes a pan mechanism and a tilt mechanism, so that the imaging direction (imaging field of view) of the attached imaging device can be changed in the vertical direction and the horizontal direction within a predetermined angle range. .
The remote controller 5-1 is provided for the user to remotely control the operation of the video conference terminal device 1-1.

In this case, the personal computer 6 and the pen tablet 7 are provided so as to be connected to the video conference terminal device 1-1.
For example, a display-format image signal corresponding to an image as a UI (User Interface) is output from the personal computer 6 to the video conference system 1-1.
The pen tablet 7 is one of pointing devices, and can perform various operations such as bringing the pen close to and touching the tablet surface. An operation signal corresponding to the pen operation performed on the tablet of the pen tablet 7-1 is input to the video conference terminal device 1-1.

  Similarly, in the place B, an imaging device 2-2, a pan head 3-2, a display device 4-2, and a remote controller 5-2 are provided together with the video conference terminal device 1-2. However, here, although the personal computer 6 and the pen tablet 7 are not shown on the video conference terminal device 1-2 side, in reality, the personal computer 6, And it is good also as a structure which provided the pen tablet 7 and was connected.

In the following description, the video conference terminal device, the imaging device, the pan head, the display device, the remote controller, and the like are not particularly distinguished from each other in the case where it is not necessary to distinguish the same devices at remote locations. 1, imaging device 2, pan head 3, display device 4, remote controller 5, and the like.
The voice transmission / reception system is not shown here.

The basic operation of the video transmission / reception system shown in this figure is as follows, for example.
First, at a location A, an image signal (captured image signal) as a moving image obtained by imaging by the imaging device 2-1 is input to the video conference terminal device 1-1. The video conference terminal device 1-1 transmits the input image signal to the video conference terminal device 1-2 via a communication line. The video conference terminal device 1-2 receives the captured image signal transmitted as described above, and outputs it from the display device 4-2.
Similarly, an image signal obtained by imaging by the imaging device 2-2 in the place B is transmitted to the video conference terminal device 1-1 by the video conference terminal device 1-2. The video conference terminal device 1-1 outputs the received image signal from the display device 4-1.
In this way, the video transmission / reception system of the video conference system performs two-way video communication, so that, for example, a conference participant in one certain place can see the state of the conference hall in another place. It can be grasped visually.
Note that the video conference terminal device 1-1 can also cause the display device 4-1 to display and output the image of the captured image signal input from the imaging device 2-1, by switching the internal signal.

  In the following, the system composed of the video conference terminal device 1-1 on the location A side and its peripheral devices shown in FIG. 1 is treated as “near end”, and the video conference terminal device 1-2 on the location B side and its In some cases, a system including peripheral devices is treated as a “far end”.

  FIG. 2 shows a video conference system (video conference terminal device and its peripheral devices) provided in one place together with an internal configuration example of the video conference terminal device 1. In correspondence with FIG. 1, the video conference terminal apparatuses 1-1 and 1-2 may be considered as having the configuration shown in FIG. 2 in common. Also in this figure, the illustration of the part corresponding to the voice transmission / reception system is omitted.

The imaging apparatus 2 in this case includes an optical system for imaging and a signal processing unit that generates an image signal as an uncompressed moving image using imaging light. An image signal (captured image signal) output from the imaging device 2 is input to the image signal processing / display control unit 11 in the video conference terminal device 1.
Further, corresponding to the optical system of the image pickup apparatus 2, a zoom mechanism unit 2a including a mechanism for changing the angle of view by moving the position of the zoom lens is provided.
The pan head 3 includes a pan mechanism unit 3a that is a mechanism for changing the imaging field of view of the attached imaging device 2 to the left and right, and a tilt mechanism unit 3b that is a mechanism for changing the imaging field of view up and down. It has.

  The image signal processing / display control unit 11 performs predetermined signal processing on the input image signal as necessary, and performs display control for causing the display device 4 to display an image.

  For example, the image signal processing / display control unit 11 performs necessary signal processing on the captured image signal input from the imaging device 2 as necessary, and transfers the processed signal to the encoder 12.

  The encoder 12 performs image compression encoding processing by a predetermined method on the input image signal, and transfers the image signal to the communication unit 14. The communication unit 14 transmits the compressed image signal transferred from the encoder 12 to another video conference terminal device via a communication line according to a predetermined communication method in accordance with the control of the control unit 18.

The compressed image signal transmitted from another video conference terminal device is received by the communication unit 14 and transferred to the decoder 13.
The decoder 13 performs demodulation (decoding) processing for image compression coding on the input compressed image signal to obtain an image signal in a predetermined format. The image signal thus obtained is transferred to the image signal processing / display control unit 11.

The image signal processing / display control unit 11 can execute display control for causing the display device 4 to display the image of the image signal transferred from the decoder 13 as described above, for example. As a result, images taken in other conference halls are displayed on the display device 4.
Further, the image signal processing / display control unit 11 can also perform display control so that the captured image signal is displayed in a predetermined manner on the display device 4.
Further, the image signal processing / display control unit 11 can execute image generation processing and display control processing for displaying various operation images on the display device 4 under the control of the control unit 18.
The image signal processing / display control unit 11 can be configured by a DSP (Digital Signal Processor) for a part or all of the functions thereof.

  In addition, the video conference terminal device 1 has a function of variably controlling the imaging field of view of the imaging device 2. Here, the imaging field of view is variable by a combination of pan control, tilt control, and zoom control. Pan control and tilt control are controls that change the imaging direction in the left-right direction and the up-down direction, respectively. Correspondingly, the video conference terminal device 1 includes a PTZ (PTZ: Pan (Panning), Tilt (Tilting), Zoom (Zooming)) drive control unit 15.

The PTZ drive control unit 15 includes a pan / tilt drive control unit 16 and a zoom drive control unit 17 as illustrated.
The pan / tilt drive control unit 16 generates a drive signal for driving the pan motor in the pan mechanism unit 3a that is supposed to be provided in the pan head 3 in accordance with the movement amount instruction information in the pan direction from the control unit 18. Output. Similarly, a drive signal for driving the tilt motor in the tilt mechanism unit 3b provided in the pan head 3 is generated and output according to the movement amount instruction information in the tilt direction from the control unit 18. As a result, the imaging field of view of the imaging device 2 can be changed by changing the imaging direction in the horizontal direction (pan direction) and the vertical direction (tilt direction).
Further, the zoom drive control unit 17 generates a drive signal for driving the zoom motor in the zoom mechanism unit 2 a provided in the optical system of the imaging device 2 in accordance with the instruction information of the angle of view change from the control unit 19. And output. This makes it possible to change the angle of view as the imaging field of view.

  The control part 18 is comprised by the microcomputer etc. which have CPU, ROM, RAM etc., for example, and performs the whole control about the video conference terminal device 1. FIG.

  The storage unit 19 is composed of, for example, a non-volatile storage device, and stores various programs and data necessary for the control unit 18.

  The remote controller 5 is provided with necessary operators for the user to perform various operations regarding the operation of the video conference terminal device 1, and in response to operations on these operators, operation information signals are transmitted by infrared rays or radio waves. Send. The operation information signal is received and demodulated by the receiving unit 20 and input to the control unit 18. The control unit 18 executes necessary control and processing in response to the input operation information signal.

In this case, the video conference terminal device 1 includes a PC image input terminal 21. By connecting the PC image input terminal 21 and the external video output terminal of the personal computer 6, the video conference terminal device 1 can input an image signal (PC image signal) output from the personal computer 6. In this case, the PC image signal input in this way is input to the image signal processing / display control unit 11.
For example, in correspondence with FIG. 1, the video conference terminal device 1-1 at the near end is connected to itself for the PC image signal input to the image signal processing / display control unit 11 under the control of the control unit 18. It can be displayed as an image on the display device 4-1. Moreover, it is also possible to transmit to the video conference terminal device at the far end in the same manner as the captured image signal input from the imaging device 2-1.
When the captured image signal and the PC image signal are input, the video conference terminal device 1-1 may select and transmit only one of the image signals when transmitting the image signal. It is also possible to transmit both image signals simultaneously using a transmission technique such as time division multiplexing. In this case, on the video conference terminal device 1-2 side that is the far end that receives the image signal, the captured image signal and the PC image signal can be received and acquired at the same time. Accordingly, the images of both image signals can be simultaneously displayed in a predetermined manner on the screen of the display device 4-2, or only one of them can be selected and displayed.

  In addition, the video conference terminal device 1 includes a tablet terminal 22. By connecting the pen tablet 7 to the tablet terminal 22, an operation signal (pen tablet operation signal) from the pen tablet 7 is input to the video conference terminal device 1. For example, the control unit 18 executes a required control process in accordance with the input pen tablet operation signal in accordance with the pen tablet-compatible application activated at that time.

  Instead of inputting the PC image signal from the personal computer 6 via a wired interface corresponding to the PC image input terminal 21, for example, the PC image signal is input via the interface via some wireless interface. Good. Similarly, on the pen tablet 7 side, instead of a wired interface corresponding to the tablet terminal 21, a pen tablet operation signal may be input via a wireless interface.

  Here, as one of the general applications compatible with the pen tablet described above, there is an application for drawing a line drawing or the like corresponding to the locus of the pen operation on the tablet. In such an application, control processing is performed so that an image drawn according to position information indicated by an input pen tablet operation signal is displayed on an image displayed on the display device 4 connected thereto. To be executed. Alternatively, the control process is executed so as to synthesize an image drawn according to the pen tablet operation signal with an image to be transmitted to the far-end video conference terminal device.

Then, for example, in the environment where the personal computer 6 and the pen tablet 7 are connected to the near-end video conference terminal device 1-1 as in the system configuration shown in FIG. It can be a way of proceeding.
That is, data serving as material is stored in the personal computer 6, and an image (PC image signal) in a state where the data of the material is opened and displayed is input to the video conference terminal device 1-1. The video conference terminal device 1-1 causes the display device 4-1 connected to itself to display and output the PC image signal at this time as an image. Further, this PC image signal is transmitted to the far-end video conference terminal device 1-2, and the received PC image signal is displayed as an image on the video conference terminal device 1-2 side by the display device 4-2. Display output. Thereby, the image of the same material opened by the personal computer 6 is displayed on both the display device 4-1 on the near end side and the display device 4-2 on the far end side. That is, all the conference participants in places A and B can see images of the same material.

  In addition, depending on the operation of the pen tablet 7, it is set so that drawing according to the movement path of the pen on the tablet can be performed on the image displayed on the display screen unit 40 input at that time. To. Thus, for example, when a meeting participant on the site A operates the pen tablet 7 while viewing the image of the material displayed on the display device 4-1, for example, the material is displayed. A necessary line drawing can be appropriately written on the image. In this way, rather than simply looking at the images of the materials and then proceeding with the meeting by relying solely on the conversation, the understanding of the materials by both the meeting participants at locations A and B, It is expected that the will will communicate smoothly.

However, since the pen operation for drawing using the pen tablet 7 as described above is relatively complicated, for example, it takes a certain amount of time and tends to increase the number of erroneous operations. In many cases, it is actually impeding the smooth progress of the conference. The above drawing operation is so-called freehand writing. However, freehand writing by the operation on the pen tablet 7 may perform drawing as intended unless the user is used to some extent. There is also the problem that it is difficult.
In addition, the video conference terminal device 1 simply needs to execute other processes related to voice and communication. However, the drawing process according to the operation of the tablet as described above is suitable as an image process. Therefore, it may be preferable to make the process lighter if possible.

In view of this, the present embodiment proposes an application corresponding to the pen tablet 7 installed in the video conference terminal device 1 with the following drawing function.
First, with reference to FIG. 3 and FIG. 4, what the drawing function corresponding to the pen tablet of this embodiment is is explained.

3A shows an image based on the PC image signal (PC display image 41) and an image based on the captured image signal from the imaging device 2 (captured display image 42) in the display screen section 40 of the display device 4. FIG. ) Are displayed simultaneously.
Note that, as the captured display image 42, an image of a captured image signal transmitted from the communication partner side is displayed and output. That is, an image captured on the communication partner side is displayed.

Also, the image content in the display screen section 40 shown in FIG. 3 and FIG. 4 including FIG. 3A described above is displayed on the display device 4-1 on the near end side.
Therefore, according to the correspondence with FIG. 1 described above, the PC display image 41 is that of the PC image signal from the personal computer 6 that is input to the same near-end video conference terminal device 1-1. The captured display image 42 is a captured image signal captured by the far-end imaging device 2-2 and transmitted from the video conference terminal device 1-2.

However, the PC display image 41 and the captured display image 42 are also displayed in the form shown in this figure on the display screen section 40 of the display device 4-2 on the far-end video conference terminal device 1-2 side. It shall be.
The PC display image 41 in this case is that of the PC image signal transmitted from the near-end video conference terminal device 1-1 that is the communication partner, and the captured display image 42 is also the image of the near-end that is the communication partner. The captured image signal is captured by the device 2-1 and transmitted from the video conference terminal device 1-1.
In this way, as described above, with respect to the PC display image 41, an image having a common content is displayed on the near-end display device 4-1 and the far-end display device 4-2. ing.

  Here, it is assumed that a conference participant in the place A (near end side) wants to write something that serves as a mark when explaining the contents of the PC display image 41, for example. In such a case, the participant operates the pen tablet 7 (or the remote controller 5-1) to display a mark image selection window on the display screen unit 40 as shown in FIG. 43 is displayed.

The mark image selection window 43 may be displayed only on the display device 4-1 connected to the near-end video conference terminal device 1-1 to which the pen tablet 7 is connected.
The mark image selection window 43 is configured such that a predetermined number of mark image selection buttons 44 are arranged and displayed, for example. Each of the mark image selection buttons 44 corresponds to a mark image having a different predetermined pattern, figure, or design. In this case, the leftmost mark image selection button 44 corresponds to the mark image of the X shape, the second mark image selection button 44 from the left corresponds to the mark image of the circle symbol, and the third mark image from the left. The mark image selection button 44 corresponds to a mark image of a symbol of Δ, and the fourth mark image selection button 44 from the left corresponds to a mark image of a symbol of □. Each mark image selection button 44 has a corresponding mark image symbol arranged and displayed.
In addition, a selection instruction frame 45 is arranged and displayed for any one of these mark image selection buttons 44. The mark image selection button 44 in which the selection instruction frame 45 is arranged indicates that it is currently a selection decision candidate.

  For example, the operator of the pen tablet 7 can move the selection instruction frame 45 with respect to the mark image selection button 44 corresponding to a desired mark image pattern by a predetermined operation on the pen tablet 7. In this case, the operator of the pen tablet 7 displays the second circle symbol from the left as shown as the transition of the display of the mark image selection window from FIG. 3 (b) to FIG. 4 (a). It is assumed that the selection instruction frame 45 is moved and arranged on the mark image selection button 44 corresponding to. In this state, it is assumed that a determination operation is performed by a predetermined operation on the pen tablet 7. As a result, the selection of the mark image with the symbol of ○ is confirmed.

Then, when the selection of the mark image with the symbol “◯” is confirmed as described above, the image (main image) corresponding to the display screen unit 40 is displayed as shown in FIG. In addition, a mark image 46 with a symbol of ○ is newly displayed.
Then, after the mark image 46 is displayed in this way, the mark image 46 can be moved to an arbitrary position on the display screen unit 40 by, for example, a drag operation on the pen tablet 7. For example, as shown as a transition from FIG. 4A to FIG. 4B, the mark image 46 can be arranged at a desired position in the line graph shown in the PC display image 42.

In this way, in the present embodiment, for example, a desired mark image is selected from mark images whose design, figure, and design are determined in advance using a GUI (Graphical User Interface) on the display screen unit 40. By performing the selection operation, the selected mark image (instruction graphic image) 46 can be displayed on the display screen unit 40.
With such an operation, for example, the user does not have to perform a troublesome and difficult operation of drawing a mark by writing freehand with the pen tablet 7.

  Although the mechanism for that will be described later, the mark image 46 is also displayed on the display screen 40 of the display device 4-2 on the far-end video conference terminal device 1-2 side. It is displayed with the same symbol and position as that displayed on the display screen 40 of the display device 4-1 on the one side. Therefore, for example, in response to the selection determination operation performed on the mark image selection button 44, the display image 42 of the far-end display device 4-2 has the same pattern and position as in FIG. A mark image 46 is newly displayed. When the display is changed as shown in FIG. 4A to FIG. 4B in the display device 4-1 on the near end side in accordance with the operation on the pen tablet 7, the same display change occurs. It also appears in the far-end display device 4-2.

In the present embodiment, for example, a plurality of mark images 46 are displayed on the display screen section 40 by the selection determination operation for the mark image selection button 44 described with reference to FIGS. 3B and 4A. The arrangement is displayed, and each of these display positions can also be moved by a drag operation. At this time, a plurality of mark images 46 having the same design can be arranged and displayed.
Further, it is assumed that the mark image 46 displayed on the display screen unit 40 can be arbitrarily selected and deleted by a predetermined operation on the pen tablet 7.

FIG. 5 shows an example of the configuration of the video conference terminal device 1 corresponding to the display of the mark image described above. In addition, the structure shown in this figure shall also have the video conference terminal apparatus 1-2 of a far end side with the video conference terminal apparatus 1-1 of a near end side.

The control unit 18 has a mark image processing block 30 as shown in the figure as a processing function obtained by executing a program, for example. The storage unit 19 stores mark image data 19a.
Further, the mark image processing block 30 inputs a pen tablet operation signal from the pen tablet 7 and executes processing related to the mark image display. That is, the processing related to the GUI as the mark image selection window 43 shown in FIGS. 3B and 4A and the processing related to the display of the mark image 46 shown in FIGS. 4A and 4B are executed. .
Further, the mark image processing block 30 here uses the mark image data 19 a stored in the storage unit 19 for displaying the mark image 46. The mark image data 19a is composed of data representing a mark image of a predetermined symbol. The mark image processing block 30 performs a process of drawing a mark image of a pattern that is designated according to the selection determination operation on the mark image selection button 44 by using the mark image data 19a with reference to the mark image data 19a. Can do.
The mark image processing block 30 generates, for example, image data of a GUI image as the mark image selection window 43 and image data as the mark image 46, which are generated by a drawing process, in the image signal processing / display control unit 11. To the unit 31. At this time, the mark image processing block 30 also designates the display position on the main image for the image represented by the output image data to the image signal processing / display control unit 11. .

  In addition, the mark image processing block 30 appropriately controls the communication unit 14 to transmit and receive display instruction information to and from other communication terminal devices, for example, as described later.

In this case, the image composition processing unit 31 composes an image obtained by compositing the PC display image 41, the captured display image 42, the mark image selection window 43, and the mark image 46 in correspondence with the display mode examples shown in FIGS. The image signal processing (image composition processing) for enabling display is performed.
As shown in this figure, the image composition processing unit 31 is input with the captured image signal, the PC image signal, the image data of the mark image selection window 43, and the image data of the mark image 46. Here, in the case of the near-end video conference terminal device 1-1, the PC image signal is input from the personal computer 6 connected to itself. In contrast, in the far-end video conference terminal device 1-2, the PC image signal of the personal computer 6 obtained by receiving the signal transmitted from the near-end video conference terminal device 1-1 is obtained. .

The image composition processing unit 31 synthesizes the image signal and the image data input as described above by, for example, digital image processing, thereby generating a display image in the mode illustrated in FIGS. 3 and 4, for example. . Then, the display image signal for one screen after the synthesis processing is output to the display device 4. As a result, the images shown in FIGS. 3 and 4 are actually displayed on the display screen section 40.
For confirmation, in this case, the main image on which the mark image 46 is to be displayed is 1 corresponding to the entire display screen unit 40 including at least the PC display image 41 and the captured display image 42. The image is displayed in the effective image area for the screen.

FIG. 6 shows a procedure example of operations executed by the near-end video conference terminal device 1-1 and the far-end video conference terminal device 1-2 for displaying the mark image according to the present embodiment. Yes. Note that the processing shown in this figure is based on the assumption that the mark image selection window 43 has already been displayed and output on the main image as shown in FIG. 3B, for example.
The procedure shown in this figure is mainly executed by the mark image processing block 30 of the control unit 18 in the near-end video conference terminal device 1-1 and the far-end video conference terminal device 1-2. You can see.

First, in the video conference terminal device 1-1 on the near end side, an operation from the pen tablet 7 corresponding to an operation on the mark image selection window 43 in response to the display and output of the mark image selection window 43. It is supposed that a signal is input and processed. Then, in step S101, a mark image selection determination operation, that is, a selection determination operation for the mark image selection button 44 in the mark image selection window 43 described with reference to FIGS. 3B and 4A is performed. It is determined whether or not it has been received.
If a positive determination result is obtained in step S101, the procedure of steps S102 and S103 is executed, and then the process proceeds to step S104. On the other hand, if a negative determination result is obtained, the procedure of steps S102 and S103 is skipped and the process proceeds to step S104.

  In step S102, display instruction information for the mark image selected and determined corresponding to step S101 is newly generated. This display instruction information (control information) includes an image ID, symbol information, and position information. Then, the newly generated display instruction information is held in, for example, a RAM or the like that the control unit 18 has for the mark image display control (step S110) in itself.

  The image ID is an identifier (number) assigned so as to be unique corresponding to the mark image to be displayed on the main screen in response to the selection and determination corresponding to the current step S101. It will be. Therefore, for example, if a plurality of mark images 46 are displayed even with the same design, different image IDs are assigned to the mark images 46, respectively. After the procedure of the near-end video conference terminal device 1-1 shown in FIG. 6 is started, each time step S102 is executed, it is generated in the past according to a predetermined rule as one of the display instruction information generation processes. An image ID that does not overlap with the generated image ID is generated.

  Further, the symbol information is information indicating what the symbol image is to be displayed on the main screen in response to the selection and determination corresponding to the current step S101. For example, if it corresponds to the example shown in FIG. 3 and FIG. 4, four symbols of x, ○, Δ, and □ are prepared in advance as the symbol image. Therefore, the symbol information indicates which symbol out of these four symbols. Note that the entity of the symbol information may be, for example, image data itself in a format that can be used for drawing an actual mark image, or may be described in order to execute a process for drawing a corresponding mark image. May be a programmed program, script, command, or the like.

  The position information is information indicating a position where the corresponding mark image 46 is to be arranged and displayed on the image as the main screen.

  In step S103, control for causing the far-end video conference terminal device 1-2 to transmit the display instruction information newly generated in step S102 is executed.

  In step S104, as an operation performed on the pen tablet 7, a drag operation for moving the mark image 46 currently displayed on both the near end side and the far end side on the main image is performed. It is determined whether or not the operation has been performed. Here, if a positive determination result is obtained, the procedure of steps S105 and S106 is executed and then the process proceeds to step S107. If a negative determination result is obtained, the process proceeds to step S107 as it is.

In step S105, the position information in the display instruction information for the mark image to be moved is updated in accordance with the moving operation corresponding to the current step S104. That is, based on the amount of change in coordinates indicated by the operation signal corresponding to the movement operation of the mark image 46, position information where the mark image 46 should be displayed is newly obtained, and the position information thus far is obtained based on the position information. Is to be overwritten.
In step S106, the display instruction information whose position information has been updated in step S105 is transmitted to the far-end video conference terminal device 1-2. At this time, instead of transmitting the updated display instruction information itself, the updated position information is transmitted together with a command for notifying that the position information in the designated display instruction information has been updated. Also good. At this time, an image ID can be used as information for specifying the display instruction information whose position information has been updated. As can be understood from the above description, the image ID is an individual identifier for each mark image that is currently displayed, and accordingly, a display instruction generated corresponding to each mark image 46 to be displayed. This is because each information is unique.

  In step S107, it is determined whether or not an operation for erasing a mark image that is currently displayed as an operation on the pen tablet 7 has been performed. If a positive determination result is obtained here, the procedure of steps S108 and S109 is performed and then the process proceeds to step S110. If a negative determination result is obtained, the process proceeds to step S110 as it is.

In step S108, for example, position information about the mark image 46 that is the object of the erasure operation determined in step S107 is deleted from the display instruction information currently held in the RAM or the like.
At the same time, in step S109, a command requesting deletion of the display instruction information corresponding to the mark image 46 that is the object of the erasure operation determined in the same step S107 is transmitted. Also at this time, the image ID can be used to indicate the display instruction information that is the target of the deletion request.

In step S110, drawing processing and display control for displaying and outputting the mark image 46 are executed based on the contents of each display instruction information currently held in the RAM, for example.
That is, for example, the mark image processing block 30 reads, from the mark image data 19a, the data for displaying the mark image corresponding to the symbol indicated by the symbol information included in the display instruction information corresponding to one display instruction information. And the drawing process is executed based on this data. Then, the image composition processing unit 31 performs image composition so that the data of the partial image as the mark image generated by the drawing process is composed at the position in the main image indicated by the position information included in the same display instruction information. Control the process.
Such drawing processing and display control processing are executed for each display instruction information currently held. As a result, display of the mark image corresponding to the user operation can be obtained.
In the flow of the procedure shown in this figure, even if no display instruction information is held, the procedure proceeds to step S110. In this case, however, in step S110, the mark image display is performed. Since it is recognized that the necessary display instruction information does not exist, substantial control for displaying the mark image is not executed.
If the procedure of step S110 is executed, the process returns to step S101.

Next, a procedure executed by the mark image processing block 30 of the far-end video conference terminal device 1-2 will be described.
On the far end side, first, in step S201, newly generated display instruction information (new display) transmitted in response to execution of steps S103, S106, and S109 of the near-end video conference terminal device 1-1. (Instruction information), updated display instruction information (update display instruction information), and waiting for reception of any information of deletion request. If any of these pieces of information is received by the communication unit 14 and passed to the mark image processing block 30, an affirmative determination result is obtained in step S201, and the procedure after step S202 is performed. Proceed to

In step S202, it is determined whether the information received corresponding to step S201 this time is new display instruction information, update display instruction information, or a deletion request.
If the received information is new display instruction information, the process proceeds to step S203. If the received information is update display instruction information, the process proceeds to step S204. If the received information is a deletion request, the process proceeds to step S205.

  In step S203, the received new display instruction information is held in, for example, a RAM.

  In step S204, among the display instruction information currently held, the display instruction information having an image ID that matches the image ID included in the received update display instruction information is rewritten to the content of the received update display instruction information. And update. However, as described above, in this case, the content updated in the update display instruction information is only the position information. Therefore, in the actual processing, only the position information is displayed from the display instruction information. You may make it update.

  In step S205, the display instruction information having the image ID indicated by the received deletion request is deleted from the display instruction information currently held.

  In step S206, similarly to step S110 described above, control for displaying the mark image 46 based on the display instruction information held in the far-end video conference terminal device 1-2 is executed. Here, the display instruction information held by the far-end video conference terminal device 1-2 for use in the procedure of step S206 is the near-end video conference terminal device 1-1 at the same timing. This is the same as the display instruction information that is held at. As a result, the same mark image can be displayed at the same position on the display devices 4 on the near end side and the far end side.

Further, depending on the above processing, when displaying the same mark image at the same position on the display device 4 on the near end side and the far end side, the image of the mark image displayed on the main image on the near end side is displayed. This is realized by sharing the same display instruction information by transmitting and receiving display instruction information or a command (deletion notification) instead of transmitting the image signal to the far end side. In this case, the drawing process and the display control process of the mark image are performed independently in each of the near-end and far-end video conference terminal devices. Thus, for example, as described with reference to FIGS. 3 and 4, even if the PC display image 41 displayed on the display device 4 on the near end side and the far end side is common, the captured display image 42 is Even when the display contents are different, such as when the captured images of the communication partners are displayed, for example, as long as the display size and display position of the PC display image 41 are the same, the mark image 46 is displayed. On the PC display image 41, they can be displayed at the same position.
In addition, in the present embodiment, since the display instruction information has the image ID, it is possible to determine the identity between the display instruction information shared on the near end side and the far end side. Thereby, it is possible to obtain consistency in the content of the display instruction information to be updated, and it is possible to delete the same mark image on the near end side and the far end side.

  In the description so far, the position information of the display instruction information deals with the display image area corresponding to the entire display screen unit 40 as the main image. For example, a display screen such as a PC display image 41 is displayed. It is also possible to treat the image portion that is a part of the display image area corresponding to the entire unit 40 as the main image, and to obtain position information that indicates the position in the main image as the partial image. Based on such position information, the mark image processing block 30 displays the display position of the mark image 46 on the partial image displayed as the main image in, for example, a display image area corresponding to the entire display screen unit 40. It is only necessary to execute a control for obtaining one coordinate and arranging and displaying the mark image 46 at the coordinates thus obtained. With such a configuration, for example, even in a situation where the PC display image 41 is displayed at different positions and sizes on the near end side and the far end side, the same position on the PC display image 41 of each other. It is possible to display the mark image 46. At this time, it is preferable to perform display control so that the display size of the mark image 46 is also changed in accordance with the size of the PC display image 42.

  For confirmation, the main image in the present embodiment is a mark image regardless of whether it is an image display area corresponding to the entire display screen section or a partial image area thereof. Is an image area that is to be arranged and displayed.

The operation for newly displaying the mark image of the present embodiment is, for example, a button selection / determination operation for the mark image selection window 43 displayed on the display screen unit 40 as described above. Yes, it is very simple.
From this, the operation of displaying the mark image according to the present embodiment is not limited to the drawing such as the pen tablet 7, but for example, an operation device capable of only the operation with respect to the operator is used. It can be said that this is also possible. For example, if the configuration of the present embodiment shown in FIGS. 1 and 2 is used, it can be said that the remote controller 5 can be used for displaying the mark image.

  Therefore, as a modified example, a configuration for displaying a mark image in accordance with an operation on the operator of the remote controller 5 instead of the pen tablet 7 is shown in FIG. In this figure, the same parts as those in FIG.

In FIG. 7, the mark image processing block 30 is configured to input a remote control operation signal output from the receiving unit 20 in place of the tablet operation signal from the pen tablet 7.
For example, if the user performs an operation for displaying the mark image selection window 43 as an operation on the remote controller 5, a remote control operation signal corresponding to the operation is input to the mark image processing block 30. The mark image processing block 30 executes a process for displaying and outputting the mark image selection window 43 in response to the input of the remote control operation signal. Next, under the state in which the mark image selection window 43 is displayed, for example, an operation for moving the selection instruction frame 45 is performed by an operation on the left and right keys of the remote controller 5, and a remote control operation signal corresponding to this is performed. Is input, the mark image processing block 30 executes display control so that the selection instruction frame 45 being displayed is moved to the mark image selection button 44 arranged in the direction corresponding to the operation. Then, in response to the input of a remote control operation signal corresponding to the determination operation on the remote controller 5, the mark image processing block 30 executes the procedure of steps S102, S103, and S110 in FIG. Further, depending on whether the mark image 46 being displayed is moved or erased as an operation on the remote controller 5, together with the procedures of steps S 105 and S 106 and steps S 108 and S 109. Step S110 is executed.

  In the case where the pen tablet 7 is used to display the mark image as in the previous embodiment, for example, the pen tablet 7 may be specialized for the video conference terminal device 1. For example, a sheet of a picture corresponding to the mark image selection button may be pasted on the tablet surface of the pen tablet 7, and the mark image selection may be determined by a pointing operation on the sheet.

  In the above description, an operation for moving the position of the currently displayed mark image is possible. For example, not only that, it is also possible to configure such that an operation of changing a symbol in a state of being displayed as it is without erasing a mark image being displayed once. In this case, the symbol information in the display instruction information may be rewritten and updated according to the operation.

In practice, three or more video conference terminal devices 1 communicate with each other to transmit and receive image signals obtained by imaging, and each video conference terminal device 1 executes display output of received images. In many cases, a video conference system is constructed. In this embodiment, in order to simplify the description so far, it is assumed that a video conference system is constructed by two video conference terminal devices 1, but the configuration based on the present embodiment is as follows. Even when a video conference system is constructed by three or more video conference terminal devices as described above, the present invention can be applied without any problem.
Further, for example, the configuration of the video conference system shown in FIGS. 1 and 2 or the internal configuration of the video conference terminal device 1 may be changed as appropriate.

  Further, the processing procedure shown in the flowchart of FIG. 6 is mainly executed by the mark image processing block 30 of the control unit 18, but this is because the CPU forming the control unit 18 executes the program. It can be seen that it is obtained by This program is, for example, a pen tablet driver or an application corresponding to the pen tablet. Such a program is written and stored in the ROM or the storage unit 19 forming the control unit 18 at the time of manufacture or the like, and is stored in a removable storage medium and then installed from the storage medium ( It is conceivable to store the information in the storage unit 19 or the like so as to include the update. It is also conceivable that the program can be installed via a data interface under the control of another host device. Furthermore, it is also conceivable that the information is stored in a storage device in a server or the like on the network and can be downloaded and acquired from the server via the network, for example.

It is a figure which shows the structural example of the video conference system corresponding to embodiment of this invention. It is a figure which shows the internal structural example of the video conference terminal device of this Embodiment with the imaging device, display apparatus, etc. which are connected with this. It is a figure for demonstrating the example of the aspect of the mark image display of embodiment, and the operation procedure example for it. It is a figure for demonstrating the example of the aspect of the mark image display of embodiment, and the operation procedure example for it. It is a figure which extracts and shows the structure for the mark image display in the video conference terminal device of embodiment. It is a figure which shows the example of a procedure of the operation | movement which the near-end side and the far-end video conference terminal device perform for the mark image display of embodiment. It is a figure which extracts and shows the composition for the mark image display in the video conference terminal device of an embodiment (modification).

Explanation of symbols

  1 (1-1, 1-2) video conference terminal device, 2 (2-1, 2-2) imaging device, 2a zoom mechanism unit, 3a pan mechanism unit, 3b tilt mechanism unit, 4 (4-1, 4 -2) Display device, 5 (5-1, 5-2) Remote controller, 6 Personal computer, 7 Pen tablet, 11 Image signal processing / display control unit, 12 Encoder, 13 Decoder, 14 Communication unit, 15 PTZ drive control 16 pan / tilt drive control unit 17 zoom drive control unit 18 control unit 19 storage unit 20 reception unit 21 PC image input terminal 22 tablet terminal 30 mark image processing block 31 image composition processing unit 40 display screen section, 41 PC display image, 42 captured display image, 43 mark image selection window, 44 mark image selection button, 45 Selection instruction frame

Claims (5)

Image signal input means for inputting an image signal;
Communication means for performing communication with other communication terminal devices;
An operation means for performing an operation of selecting an instruction graphic image to be displayed from a predetermined number of instruction graphic images whose modes are determined;
The image signal input by the image signal input means is displayed and output as a main image or an image portion thereof, and when the instruction graphic image is selected by an operation on the operation means, the selected instruction signal is displayed. Display control means for controlling the figure image to be synthesized and displayed in the main image; and
An image transmission means for causing the communication means to transmit the image signal input by the image signal input means as an image signal to be displayed and output as a main image or an image portion thereof on the other communication device side;
In response to the instruction graphic image being selected by the operation on the operation means, the selected instruction graphic image is synthesized in the main image that is displayed on the other terminal device side. Control information generating means for generating control information to be displayed;
Control information transmission control means for transmitting the control information by the communication means;
A communication terminal device comprising: The control information generation means forms the control information so as to include image identification information that is unique to each instruction graphic image to be displayed by the control information.
The communication terminal device according to claim 1. An image signal input procedure for inputting an image signal;
A communication procedure for performing communication with other communication terminal devices;
The image signal input by the image signal input procedure is displayed and output as a main image or an image portion thereof, and when the instruction graphic image is selected by an operation on the operation means, the selected instruction signal is displayed. A display control procedure for controlling the figure image to be synthesized and displayed in the main image;
An image transmission procedure that causes the image signal input by the image signal input procedure to be transmitted by the communication procedure as an image signal to be displayed and output as a main image or an image portion thereof on the other communication device side;
In response to the instruction graphic image being selected by the operation on the operation means, the selected instruction graphic image is synthesized in the main image that is displayed on the other terminal device side. A control information generation procedure for generating control information to be displayed;
A transmission control procedure for transmitting the control information by the communication procedure;
The image processing method characterized by performing. Communication means for performing communication with other communication terminal devices;
Holds control information received from the communication means and transmitted from another communication terminal device to display any one of the specified number of instruction graphic images whose modes are defined. Control information holding means for
Main image corresponding display control means for displaying and outputting an image signal received by the communication means and transmitted from another communication terminal device as a main image or an image portion thereof;
Based on the control information held by the control information holding means, an instruction graphic image display control means for controlling the instruction graphic image to be synthesized and displayed in the main image;
A communication terminal device comprising: A communication procedure for performing communication with other communication terminal devices;
Control information received from the communication procedure and transmitted from another communication terminal device to display any one of the predetermined number of instruction graphic images whose modes are defined is held. Control information retention procedure to
A main image corresponding display control procedure for displaying and outputting the image signal received by the communication procedure and transmitted from another communication terminal device as a main image or an image portion thereof;
An instruction graphic image display control procedure for controlling the instruction graphic image to be synthesized and displayed in the main image based on the control information held by the control information holding procedure,
The image processing method characterized by performing.

Images