JP2015109565A - Conference terminal device, electronic conference system, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To notify a participant in a remote location of the state where an operator performs a writing operation and pointing operation to a white board, in an electronic conference system using a projector with a camera.SOLUTION: A conference terminal device detects a writing operation and pointing operation to a white board from a photographed image of the white board, and creates operation information according to the respective operations and transmits the created operation information to a conference terminal in a remote location. When the conference terminal has received the operation information, projects an image object showing the writing operation or the pointing operation on a white board according to the received operation information.

Description

  The present invention relates to a conference terminal device, an electronic conference system, and a program for sharing image information captured by a camera of each terminal among a plurality of bases via a network and displaying the information on each terminal.

  An electronic conference system may be used to hold a conference with another site at a remote location at the site. In the electronic conference system, the following processing is performed. A text or figure written on a writing object such as a whiteboard or blackboard installed at another base is photographed by a photographing device such as a camera installed at the other base. Then, a projection device such as a projector installed at its own site receives an image taken at another site and projects the image onto a whiteboard or the like installed at its own site. By performing such processing, it is possible to refer to the contents written on the writing object installed at the other site also at the own site (for example, Patent Document 1).

JP 2011-151764 A

  The electronic conference system of Patent Document 1 makes it possible to refer to the content written on a writing object such as a whiteboard installed at another base at the base. In such an electronic conference system, in addition to the contents written on the whiteboards at other sites, the position of the area currently written on the whiteboards at other sites can be referred to in real time at the local site. Is desirable. In other words, it is desirable that the user at the local site can recognize the position of the area operated by the user at the other site on the whiteboard or the like at the other site in order to hold a smoother electronic conference. However, in the invention described in Patent Document 1, the position of the area operated by the user at the base at one base cannot be recognized by the user at the other base.

  The present invention has been made in view of the above-mentioned conventional example, and is an electronic conference that can share not only the written contents but also the area operated by the user such as the ongoing writing and the position designated by the user even at a remote site. The purpose is to provide a system.

  In order to achieve the above object, the present invention has the following configuration.

A conference terminal device,
Photographing means for photographing the writing object;
Projecting means for projecting an image onto the writing object;
Control means for controlling the conference terminal device;
Communication means for communicating with other conference terminal devices,
The control means includes
Detecting movement of a person's hand from the image taken by the photographing means, and sending operation information indicating an image object corresponding to the detected hand movement to the other conference terminal device by the communication means;
An image object is generated based on the operation information received by the communication unit from the other conference terminal device, and projected by the projection unit.

  According to the present invention, it is possible to notify a state of operation on a writing object such as a whiteboard in a remote place.

The figure which shows the network configuration example of the conference system The figure which shows the hardware constitution example of the conference terminal Figure showing a software configuration example of a conference terminal Diagram showing the state when writing to the whiteboard Flowchart of processing for generating writing image information based on a photographed image The figure which shows the example which produces | generates writing image information based on a picked-up image Flowchart of processing for generating operation information based on a photographed image Diagram showing the state when pointing with a finger to the whiteboard Diagram showing analysis of whiteboard shot image and operation status , The figure which shows an example of the operation state information produced | generated from the analysis result of the operation state The figure which shows an example of the operation locus information produced | generated from the analysis result of the operation state Electronic object reception processing flowchart Flow chart of board image forming process A figure showing an example of a background image Flowchart of processing for generating operation information based on a captured image in the second embodiment The figure which shows the software configuration in Embodiment 3. Flowchart of processing for generating operation information based on a captured image and a voice recognition state in the third embodiment The flowchart of the process which produces | generates operation information based on the switching information of the picked-up image and manual operation state in Embodiment 4.

[Embodiment 1]
Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

<Network configuration>
FIG. 1 shows a network configuration example of the conference system. Conference terminal devices (hereinafter referred to as conference terminals) 101 and 102 are connected to the Internet 105, which is a communication network, via gateway servers 103 and 104, respectively. The conference terminals can transmit and receive data via the gateway servers 103 and 104 and the Internet 105. Hereinafter, the description will be given focusing on the processing performed on the conference terminal 101 side. Therefore, the conference terminal 101 is referred to as “this terminal”, and the conference terminal 102 is referred to as “remote terminal”. Remote means that it is via communication, regardless of distance. Note that this relationship is reversed if attention is paid to the conference terminal 102. Further, in the present embodiment, the number of remote terminals is not limited to one, but it will be clarified in advance that the present embodiment can be applied even when a plurality of remote terminals are connected. In addition, each device at each site first has a writing object such as a whiteboard placed on it, and a camera is installed at a position where the writing can be photographed, preferably almost in front of it. A projector is installed at a position where projection is possible.

<Hardware configuration>
FIG. 2 shows a hardware configuration example of the conference terminal. In the present embodiment, a projector with a camera is used as a conference terminal. Although the conference terminal is described here as a dedicated hardware device for connecting to the conference system, it can be configured by combining a projector and a smartphone with a camera. It can also be configured by connecting a commercially available Web camera to the projector. You can also use a camera-equipped smartphone with a projector. In any case, a camera for photographing a recording board such as a whiteboard, a projector for projecting an image received from a remote terminal, a processing device for performing image processing of an image photographed by the camera, and the like The conference terminal includes a communication device for transmitting and receiving image data and the like to and from the terminal or server.

  The conference terminal is connected to the CPU 201, the RAM 202, and the ROM 203 via the main bus 200. Further, the CPU 201 is connected to a control module such as an external device controller 204 and an internal device controller 205 via the main bus 200. The CPU 201 is a unit for executing a program 300 described later. The RAM 202 is a readable / writable memory for data required when the CPU 201 executes the program 300 (shown in FIG. 3). A ROM 203 is a read-only memory for reading data necessary for the CPU 201 to execute the program 300.

  The external device controller 204 is a module for controlling external devices such as the image capturing module 211 and the image projection module 212. The image photographing module 211 is a module for photographing an image on a photographing surface to be photographed and outputting it as image data (referred to as a photographed image). The image projection module 212 is a module for displaying the image data (referred to as a projection image) generated by the CPU 201 on the projection plane. A human interface device (HID) module 213 is a module for accepting an operation from a human. The voice input module 214 is a module for receiving voice input from a human and outputting it as voice data. For example, a CCD camera module can be used as the image capturing module 211, and a liquid crystal panel projection module can be used as the image projection module 212. Further, a mouse can be used as the HID module 213, and a microphone can be used as the voice input module 213. The HID module 213 provides a user interface to the user.

  The internal device controller 205 is a module for controlling internal devices, and is a module for controlling the internal storage device 221 and the network module 222. The internal storage device 221 is a secondary storage device for storing data and programs. The network module 222 is a module for transmitting / receiving information to / from another device on the network. As the network module 222, not only a network module supporting a wired LAN but also a network module supporting a wireless LAN can be applied.

<Software configuration>
FIG. 3 shows a software configuration example of the conference terminal. A program 300 is a program stored in the RAM 202 or the ROM 203 and executed by the CPU 201 to realize a conference system. The connection setting 301 is setting data that stores connection information such as an IP address of a remote terminal that is a connection destination. The captured image input unit 311 receives a captured image captured by the image capturing module 211. In the present embodiment, the photographed image input unit 311 performs preprocessing for geometric correction and color adjustment. The written image generation unit 312 analyzes the captured image and generates written image information based on the written image obtained by converting the physical writing drawn on the whiteboard. The written image update unit 313 updates the written image information displayed on the terminal and the remote terminal, and stores it in the electronic object storage unit 331 and transmits it to the remote site. An electronic object is particularly called an image object if it is a component included in an image. For example, written characters and images described later (also referred to as writing), image components indicating writing operations and pointing operations by an operator, and the like correspond to image objects.

  The operation information generation unit 341 analyzes the captured image and generates operation information based on the state (or result) of human detection and hand shape detection. The operation information is roughly divided into two, one is operation state information indicating the latest state of the operation information, and the other is operation locus information for leaving the operation information as a locus. The operation state storage unit 342 stores operation state information. The operation locus storage unit 343 stores operation locus information. The operation information update unit 344 updates the operation information transmitted to the terminal and the remote terminal, and stores it in the electronic object storage unit 331 and transmits it to the remote site. Based on the connection setting 301, the electronic object transmission / reception unit 314 generates a packet having the electronic object as a payload in a format in which the writing image update unit 313 and the operation information update unit 344 can transmit / receive the electronic object that needs to be transmitted on the network. To do. Then, the data is transmitted to the remote terminal via the network module 222.

  Based on the connection setting 301, the electronic object receiving unit 321 receives a packet including an electronic object transmitted from a remote terminal, and extracts the electronic object included in the packet. The electronic object update determination unit 322 updates the information stored in the electronic object storage unit 331 according to the type of electronic object extracted from the received packet by the electronic object reception unit 321. The panel image forming unit 332 forms an image from the written image information and the operation information stored in the electronic object storage unit 331. There are three types of images to be formed. The first is a writing image for extracting physical writing drawn on the whiteboard at the base from the captured image. The image for writing and photographing is an image projected when photographing the writing on the whiteboard at the base, and is, for example, a blank image so that the projected image is not photographed. The second is a panel sharing image for projecting the whiteboard writing image and operation information on the remote terminal, and the third is a background image for extracting the operation information of the base from the photographed image as a background difference. is there. Since operations on the whiteboard are performed in a state where the writing information of the remote terminal is projected and superimposed on the writing, the writing acquired by photographing the whiteboard with the own terminal, for example, to extract the operation An image for writing image recording and an image for board surface sharing, such as an image obtained by combining an image and a board surface shared image projected on the whiteboard at that time, are stored in the projection image storage unit 335, and the projection image output unit 333 A projected image is projected onto the projection plane using the image projection module 313. The background image is stored in the background image storage unit 336, and the operation information generation unit 341 uses it for the background difference.

  In addition, the gesture input unit 361 analyzes the gestures / hand gestures of the conference participants from the captured images captured by the image capturing module 211 and notifies the writing image operation unit 364 of them. The HID input unit 362 accepts an operation performed by the HID module 213 and notifies the write image operation unit 364. The voice input unit 363 analyzes an operation performed by the voice of the conference participant from the voice data input from the voice input module 214 and notifies the writing image operation unit 364 of the operation. The writing image operation unit 364 instructs the writing image information generation unit 312 to perform an operation for acquiring writing on the whiteboard as a writing image.

  Here, the program 300 may have other general functions required for the electronic conference system. For example, in order to designate a remote terminal, a connection destination setting dialog box for setting a connection destination conference terminal can be provided. In order to select a remote terminal, it is possible to provide a function of inquiring of a terminal that can be connected to a server existing outside via the network module 222 and a function of notifying the dialog box as a candidate. In order to ensure security in the conference, the electronic object packet transmission unit 314 can have a function of encrypting the electronic object packet transmitted. The electronic object packet receiving unit 321 can have a function of decoding the electronic object packet received. These are general functions used in the prior art electronic conference system, and do not directly affect the essence of the present embodiment, so the details are omitted. In FIG. 3, all or a part of each block included in the program 300 described as the program module may be realized by hardware.

  Hereinafter, as the hardware, the CPU 201 executes the writing image information generation unit 312, the operation information generation unit 341, the received electronic object update execution determination unit 322, and the board image formation unit 332. explain.

<Writing image information and operation information>
Using FIG. 4 and FIG. 8, writing image information and operation information handled as an electronic object in this embodiment will be described in association with a whiteboard. FIG. 4 shows a state when writing to the whiteboard. FIG. 8 shows a state when the finger is pointed to the whiteboard. 4 (a) to 4 (e) show how the whiteboard is written at the terminal 101 base, the operation state during the writing is displayed on the terminal 102, and the written image of the terminal 101 is displayed on the terminal 102. The situation is shown in order. Here, the terms are first defined.

Written image: An image obtained by shooting a whiteboard and converting writing with a marker pen into an image.
Operation state (writing): Indicates the state of writing. For example, the state of gripping the pen from the photographed image is detected as the operation state, and the generated operation state information is sent to the remote terminal. On the remote terminal side that has received the operation state information, the operation state (writing) is displayed with a silhouette of the person and a circle around the hand.
Operation locus: A locus of a writing operation on the whiteboard, which indicates that the operation state (writing) has been maintained at a certain distance for a certain time. The display of the operation locus by the remote terminal is held until the next timing of sending the written image.

  FIG. 4A shows an initial state in which nothing is written on the whiteboard of the main base (terminal 101) and the remote base (terminal 102). In this state, the base terminal at each base is activated, and execution of the program 300 in FIG. 3 is started. However, the program 300 may be executed in a state where writing to the whiteboard has already been performed.

  When writing 402 to the whiteboard on the terminal 101 side with a marker pen is started, an operation locus 403 indicating the writing range to the whiteboard and an operation state 404 indicating the current operation state are projected as images on the whiteboard on the terminal 102 side. And displayed. FIG. 4B shows this state.

  When writing to the whiteboard on the terminal 101 side is continued, the operation locus 403 spreading rightward and the current operation state 404 moved rightward are projected and displayed on the whiteboard terminal 102 side as images. . FIG. 4 (c) shows this state.

  In the state of FIG. 4 (c), even if the writing 402 to the whiteboard on the terminal 101 side is stopped, the whiteboard on the terminal 102 side is operated until the image written on the whiteboard of the terminal 101 is displayed on the terminal 102. The state where the locus 403 is projected is maintained. FIG. 4 (d) shows this state.

  When the terminal 101 transmits the written image 405 of the whiteboard to the terminal 102, the received written image 406 is projected and displayed on the whiteboard on the terminal 102 side, and at the same time, the display of the operation locus is canceled and disappears. FIG. 4 (d) shows this state.

FIGS. 8A to 8C show a state where the pointing operation state is displayed on the terminal 102 when pointing with a finger to the whiteboard at the terminal 101 base. It also shows the behavior when the person's hand state cannot be detected from the captured image. First, define terms. Operation state (point instruction): Indicates a state of pointing with a finger. Operation state information generated by detecting the pointing state from the captured image is sent to the remote terminal. The remote terminal that has received the operation state information displays the operation state (point instruction) with a silhouette of the person and a circle around the hand.
Operation state (hand not detected): Indicates a state where a person is standing. If only the human region is detected from the photographed image and the hand shape cannot be detected, operation state information indicating the operation state (hand not detected) is generated, and the information is sent to the remote terminal. Only the silhouette of the person is displayed on the remote terminal side that has received the operation state information.

  FIG. 8A shows a state in which the written images are superimposed and projected on the whiteboard at the main site (terminal 101) and the remote site (terminal 102). It shows that the writing image 803 of the writing 801 “a + b =” on the terminal 101 side is projected to the terminal 102 side, while the writing image 802 of the writing 804 “2” on the terminal 102 side is projected to the terminal 101 side. ing. In this state, when a person points a finger to the whiteboard on the terminal 101 side, an operation state (point instruction) is detected, and operation state information indicating the operation state is transmitted to the terminal 102. The terminal 102 that has received the message displays an operation state (point instruction) 805 indicating the current operation state on the whiteboard. FIG. 8B shows this state. In FIG. 8C, a person is writing with the back on the whiteboard on the terminal 101 side. In this state, even if the person area can be detected, the shape of the hand cannot be detected. Therefore, the terminal 101 side detects the operation state (hand undetected) and transmits operation state information indicating it to the terminal 102. The terminal 102 that has received it projects and displays the operation state (hand undetected) 806 that is the current operation state on the whiteboard on the terminal 102 side.

<Writing image information generation process>
A process for generating write image information will be described with reference to FIGS. 5 and 6. FIG. 5 is a flowchart of a process for generating written image information based on a captured image. This process is performed by the writing image generation unit 312 and generates writing image information based on a writing image obtained by analyzing a photographed image and converting physical writing drawn on a whiteboard. FIG. 6 shows an example in which writing image information is generated based on a captured image. This process is performed for each frame of the moving image sent from the photographed image input unit 311. Alternatively, it may be performed every predetermined number of frames. FIG. 6 is executed by the CPU 201.

  It is determined from the photographed image etc. whether it is suitable for the generation timing of the written image (S4000). In the following description, the generation timing of the written image is referred to as shooting timing. A conference participant can perform an operation via the gesture input unit 361, the HID input unit 362, and the voice input unit 363, and can receive the shooting timing from the writing image operation unit 364. For example, the gesture input unit 361 analyzes a photographed image photographed by the image photographing module 211, and can determine that it is a photographing timing with a specific gesture / hand gesture performed therein. The gesture input unit 361 can also determine that it is the shooting timing with the timing when the moving object disappears from the captured image. The HID input unit 302 interprets that the mouse is clicked as the HID module 213, and it can be determined that it is the photographing timing. The voice input unit 263 interprets the voice input by the voice input module 263, and the timing at which the specific keyword is detected can be determined as the shooting timing.

  The process branches based on the result of determining whether it is the above photographing timing (S4010). If it is not the photographing timing, the processing for this frame is terminated. If it is the photographing timing, a projection instruction for the written photographing image is set (S4020). This is an instruction for projecting a writing image to the projector to a board surface image forming unit 332 described later. Then, the whiteboard on which the image for writing and photographing formed by the board image forming unit 332 is projected is photographed to obtain a photographed image (S4022). The image for writing and photographing is an image for projecting onto the whiteboard in order to photograph the written image drawn on the whiteboard, for example, a blank image.

  An example of this processing is shown in FIGS. In FIG. 6A, the image 601 of the writing 602 of the terminal 102 base is projected on the terminal 101 base. In FIG. 6B, the user writes the writing 603 on the whiteboard at the terminal 101 base. During writing, the operation locus 604 is displayed by the terminal 102 as shown in FIG. Then, at the base 101, as shown in FIG. 6C, a written image 605 is projected, and a captured image in a state where the writing of the terminal 102 is stopped is acquired. In this example, the writing image 605 is blank, and the writing image 601 of the writing 602 in the terminal 101 is not projected.

  When the photographed image is acquired in S4022, the instruction to project the written photographed image to the panel image forming unit 332 is canceled (S4023). The acquired image is subjected to image processing such as geometric transformation of the board shared area and color processing, and whiteboard writing is generated as a writing image (S4024). Write image information is generated based on the generated write image (S4026). An example of written image information is shown in FIG. As shown in the figure, the written image information 610 has attributes of object ID 611, object type 612, generation terminal information 613, generation time information 614, image information 615, and image position information 616. An object ID 611 is an identifier that is sequentially assigned to an electronic object that includes written image information. The object type 612 indicates the type of electronic object, and here, information “written image” indicating written image information is set. In the generation terminal information 613, an identifier representing the terminal that generated the electronic object is set. In this embodiment, the terminals 101 and 102 are used. Generation time information 614 indicates the time when the electronic object was generated. Image information 615 is an image data body generated as a written image. Image position information 616 is coordinate information for arranging a written image in the board surface shared area. In the present embodiment, a coordinate system having the origin at the upper left of the board surface shared area is used. The written image is an example in which the entire board shared area is always arranged at the upper left, and the image position information is (0, 0).

  The generated writing image information is stored in the electronic object storage unit 331 by the writing image information update unit 313 (S4028), and is transmitted to the remote terminal using the electronic object transmission unit 314 (S4030).

  As shown in FIG. 6D, when the generated write image information 607 is transmitted to the remote terminal, the write image information 607 written to the remote terminal (terminal 102) by the present terminal (terminal 101), that is, “a + b =” The projected image is projected and the writing on the whiteboard is shared.

<Operation information generation process>
Processing for generating operation information will be described with reference to FIGS. 7, 9, 10A, 10B, 11, and 14. FIG. FIG. 7 is a flowchart of processing for generating operation information based on a captured image. This processing is performed by the operation information generation unit 341, and operation information is generated based on the operation state of the person obtained by analyzing the captured image. FIG. 9 shows an example of the correspondence between the captured image of the whiteboard and the operation state. 10A and 10B (collectively referred to as FIG. 10) show an example of operation state information generated from the analysis result of the operation state. FIG. 11 shows an example of operation trajectory information generated from the analysis result of the operation state. FIG. 14 shows an example of a background image. Processing and a processing example will be described along the flowchart of FIG. A processing example will be described with reference to an example of operation state information generated from the operation state analysis result shown in FIG.

  In step S4100, a background image is acquired from the background image storage unit 336. FIG. 9 shows an example of the state of the whiteboard on the terminal 101 side, and FIG. 14 shows an example of the background image. As shown in FIG. 9A, the image “2” projected as “a + b =” is present in the photographing range 901 of the whiteboard. In this case, as shown in FIG. 14, the writing image information corresponding to the writing image “a + b =” acquired by the terminal 101 and the writing image “2” acquired by the terminal 102 is synthesized according to the respective position information. Then, the synthesized background image “a + b = 2” is stored in the background image storage unit 336. In this example, only the written image constitutes the background image. However, in this terminal displaying the operation state of the remote terminal on the whiteboard, the image indicating the operation state of the remote terminal is also synthesized with the background image. In step S 4102, a captured image is acquired from the captured image input unit 311. In step S4104, a background difference image is generated from the captured image and the background image. In step S4106, a person region is extracted from the background difference image. An example of the processing so far is shown in FIG. 9 (a) shows no person area detection, FIGS. 9 (b) to 9 (d) show person area detection, FIG. 9 (e) shows person area detection, and detects a plurality of person areas. It is in a state.

  In step S4110, the process branches depending on whether or not a person area is detected. Since the operation state information is generated in the subsequent steps, the format of the operation state information will be described first. Here, if an object is included in the background difference image, it is determined as a person area. For example, the position of the written image may be shifted between the captured image and the background image. Therefore, for example, an outline that cannot be completely erased included in the background difference image may be detected and correction may be performed to delete it.

  As shown in FIG. 10, the operation state information 1000 includes an object ID 1001, an object type 1002, generation terminal information 1003, generation time information 1004, person presence / absence 1005, person ID 1006, person area information 1007, person position 1008, hand detection 1009, Has operation type 1010 and hand position 1011 attributes. Here, the object ID 1001 is an identifier that is sequentially assigned to the electronic object including the operation state information. The object type 1002 indicates the type of electronic object. Here, “operation state” indicating operation state information is set. The generation terminal information 1003 uses an identifier representing the terminal that generated the electronic object. In this embodiment, the terminals 101 and 102 are used. Generation time information 1004 indicates the time when the electronic object was generated. Person presence / absence 1005 indicates whether or not a person area is detected. The person ID 1006 is an identifier that is sequentially assigned to the detected person area. Here, 1, 2, and 3 are assigned in order from the left side of the photographed image. The person area information 1007 is the image data body of the extracted person area. The person area is a rectangle and is a single color image showing the person area. The person position 1008 is position coordinate information of the person area with respect to the board surface shared area. In the present embodiment, the coordinates of the upper left corner of the person area are stored in a coordinate system with the upper left corner of the board shared area as the origin. Hand detection 1009 indicates whether or not a hand shape is detected. The operation type 1010 is an operation type determined from the hand shape detection result, and stores either “point instruction” or “writing”. The hand position 1011 indicates the detection position of the hand shape. Stores the center coordinates of the hand detection position.

  If no person area is detected in step S4110, the process advances to step S4120 to generate operation state information without a person area. This processing example corresponds to the analysis state shown in FIG. 9A and generates operation state information shown in FIG. Since an image such as a written image and writing displayed in the shooting range 901 of FIG. 9 are included in the background image, no person area is detected from the shot image of the whiteboard of FIG. 9A. Therefore, since there is no person area, “None” is set in the person presence / absence 1005, and other information regarding the person and hand detection is unnecessary (null). Proceed to step S4130.

  If a person area is detected in step S4110, the process proceeds to step S4150. Steps S4150 to S4166 are looped by the number of detected human areas. In FIG. 9B to FIG. 9D, since one person area is detected, the number of loops is one. Since FIG. 9E detects two person areas, the number of loops is two.

  In step S4150, the hand shape is template-matched to the new person area. There are three types of hand shapes to be detected here. The first is the hand shape detection shown in FIG. 9B, the second is the hand pointing shape shown in FIG. 9C, and the third is the hand shown in FIG. 9D. It is a writing shape. In order to accurately distinguish between a finger and a pen, for example, a mark with a specific identification color or shape may be attached to the pen.

  In step S4152, the process branches according to the template matching result of the hand shape. If the hand shape is not detected in step S4152, the process proceeds to step S4154, and undetected type operation state information is generated. Specifically, the operation state information 1000 shown in FIG. 10B is generated. Because there is a person area, the presence / absence of person 1005 is “Yes”, person ID 1006 is 1, person area information 1007 is the person area image of FIG. 9B, person position 1008 is the coordinate information shown in FIG. Since 1009 is not performed, “None” is set, and information regarding the operation type 1010 and the hand position 1011 is not necessary (null) is set. Proceed to step S4160.

  If it is determined in step S4152 that the hand shape is a point indication shape, the process advances to step S4156 to generate point indication type operation state information. Specifically, the operation state information 1000 shown in FIG. 10C is generated. Because there is a person area, the presence / absence of person 1005 is “Yes”, person ID 1006 is 1, person area information 1007 is the person area image of FIG. 9C, person position 1008 is the coordinate information shown in FIG. 1009 is “present”, the operation type 1010 is “point indication”, and the hand position 1011 is set to the coordinates for detecting the hand shown in FIG. Proceed to step S4160.

  If it is determined in step S4152 that the hand shape is a writing shape, the process advances to step S4158 to generate writing type operation state information. Specifically, the operation state information 1000 shown in FIG. 10 (d) is generated. Because there is a person area, the presence / absence of person 1005 is “Yes”, person ID 1006 is 1, person area information 1007 is the person area image of FIG. 9D, person position 1008 is the coordinate information shown in FIG. 1009 is “Yes”, the operation type 1010 is “Write”, and the hand position 1011 is set to the coordinates of detecting the hand shown in FIG. Proceed to step S4170. The operation state information 1000 generated by the processing so far is stored in the operation state storage unit 342.

  Next, processing related to generation of operation trajectory information will be described. The operation trajectory information is generated when it is determined that the operation state (writing) is held within a certain distance for a certain time as described with reference to FIG. The operation trajectory information generation process is performed in processes after step S4170. Further, the operation trajectory information which becomes unnecessary during the determination is deleted in the processing after step S4160.

  First, generation of operation trajectory information will be described from step S4170. The operation trajectory information has a format shown in FIG. The operation locus information 1100 is stored in the operation locus information storage unit 343. The content conforms to the operation type information writing type. That is, the items 1101 to 1111 included in the operation trajectory information 1100 conform to the items 1001 to 1011 of the operation state information 1000. The object type 1102 is different. There are two types of object types 1102: trajectory candidates and trajectory confirmation. FIG. 11A is an example of information temporarily stored as trajectory information candidates in the operation trajectory information 1100, and FIG. 11B is an example of information confirmed as trajectory information in the operation trajectory information 1100. It is.

  In step S4170, it is determined whether or not operation locus candidates, that is, operation locus information 1100 whose object type 1102 is “trajectory candidate” exists in the operation locus storage unit 343. If there is no operation locus candidate, the process proceeds to step S4182, and a new operation locus candidate is generated. Specifically, the operation state information 1000 of the writing type generated in step S4158 is duplicated, and operation locus information 1100 whose object type is changed from “operation state” to “trajectory candidate” is newly generated as a new operation locus candidate. . Proceeding to step S4166, if other person areas still remain, the process returns to step S4150. If no other person area remains in step S4166, the process proceeds to step S4130.

  On the other hand, if an operation locus candidate exists in step S4170, the process proceeds to step S4172. In step S4172, the operation trajectory candidate generation time information stored in the operation trajectory storage unit 343 is referred to, and it is determined whether a certain time has elapsed from the generation time of the current time. If the predetermined time has not elapsed since the generation time (NO), the process proceeds to step S4166 without performing any processing. If other person areas still remain in step S4166, the process returns to step S4150. If no other person area remains in step S4166, the process proceeds to step S4130.

  On the other hand, in step S4172, the generation time information of the operation trajectory candidate stored in the operation trajectory storage unit 343 found in step S4170 is referred to, and when a predetermined time has elapsed from the generation time (YES), the process proceeds to step S4174. . In step S4174, the position coordinate of the hand of the operation track candidate stored in the operation track storage unit 343 is compared with the position coordinate of the hand of the write-type operation state information generated in step S4158. The comparison is performed by calculating the distance between the two positions. If the position difference, that is, the distance is equal to or smaller than the predetermined distance (YES), the current operation locus candidate is changed to the confirmed operation locus in step S4176 and held. Specifically, the object type 1102 shown in FIG. 11A of the operation trajectory candidate found in step S4170 is changed from “trajectory candidate” to “trajectory confirmation” shown in FIG. 11B. . Proceed to step S4178. In step S4178, the operation state information 1000 of the writing type generated in step S4158 is duplicated, and the new operation locus information 1100 whose object type is changed to “trajectory candidate” is used as a new operation locus candidate as the operation locus storage unit 343. And go to step S4166.

  On the other hand, if the hand position difference determined in step S4174 is not less than a certain distance (NO), the operation trajectory information of the operation trajectory candidate is updated in step S4180. Specifically, the operation trajectory information 1100 obtained by changing the object type 1002 of the write-type operation state information generated in step S4158 to “trajectory candidate” is stored in the existing operation trajectory information (that is, the operation trajectory candidate found in step S4170). Replace with In other words, each field excluding the object type 1102 of the existing operation locus candidate is overwritten with the value of the corresponding field of the operation state information generated in step S4158. Proceed to step S4166. If other person areas still remain in step S4166, the process returns to step S4150. If no other person area remains in step S4166, the process proceeds to step S4130.

  As described above, when it is detected that the hand has a writing shape, if the hand moves at a speed equal to or less than a certain speed, it is determined that writing is being performed, and the locus of the hand is determined for a certain period of time. Record as every other coordinate.

  Next, unnecessary operation track information deletion processing will be described. In step S4160, it is confirmed whether an operation locus candidate exists in the operation locus storage unit 343. If there is no operation locus candidate (NO), the process proceeds to step S4166.

  On the other hand, if there is an operation locus candidate in step S4160 (YES), the process proceeds to step S4162. In step S4162, with reference to the generation time information of the operation trajectory candidate stored in the operation trajectory storage unit 343, it is determined whether a certain time has elapsed from the generation time. If the predetermined time has not elapsed since the generation time (NO), the process proceeds to step S4166 without performing any processing.

  On the other hand, in step S4162, the generation time information of the operation trajectory candidate stored in the operation trajectory information storage unit 343 is referred to, and when a predetermined time has elapsed from the generation time (YES), the process proceeds to step S4164. It is determined that a point indication type operation trajectory candidate for which a predetermined time has elapsed from the generation time is unnecessary. In step S4164, the operation trajectory information 1100 of the operation trajectory candidate that has become unnecessary is deleted, and the process proceeds to step S4166. If other person areas still remain in step S4166, the process returns to step S4150. If no other person area remains in step S4166, the process proceeds to step S4130.

  The update process of the operation state information and the operation trajectory information will be described in steps S4130 and S4140. This processing is processed by the operation information update unit 344. In step S 4130, the operation state information stored in operation state storage unit 342 is stored in electronic object storage unit 331. In step S4140, the operation state information stored in the operation state storage unit 342 and the confirmed operation track information stored in the operation track storage unit 342, that is, the operation track information whose object type 1102 is “track determination” are The object packet transmission unit 314 is used to transmit to the remote terminal. The transmitted operation state information and operation trajectory confirmation information are deleted, and the process ends.

<Electronic object reception processing>
The electronic object reception process will be described with reference to FIG. FIG. 12 is a flowchart of the electronic object reception process. This process is performed by the electronic object update determination unit 322, and the contents of the electronic object storage unit are updated according to the type of the received electronic object. The objects to be received are roughly classified into three types. The written image information shown in FIG. 6 (e), the operation state information shown in FIGS. 10 (a) to 10 (e), and the confirmed operation trajectory information shown in FIG. 11 (b).

  In step S4200, the electronic object received using electronic object receiver 321 is acquired. In step S4210, the type of the received object is determined based on the object type. If it is determined in step S4210 that the received object is the written image information 610, the process proceeds to step S4220, and the received written image information 610 and the generation terminal 614 are the same among the operation trajectory information stored in the electronic object storage unit 331. Is deleted, and the process proceeds to step S4230. This is because, when the written image 406 shown in FIG. 4E is received, the display of the operation trajectory 403 that has been projected and displayed is erased. In step S4230, among the written image information 610 stored in the electronic object storage unit 331, the received written image information 610 and the generation terminal 614 update the same written image information with the received written image information 610 and process it. Exit. This process completes the update of the written image information sent from the remote terminal.

  If it is determined that the object received in step S4210 is the operation state information 1000, the process proceeds to step S4240, and the operation state information stored in the electronic object storage unit 331 and the generation terminal having the same operation state information are received. Update with the operation status information and end the process. This process completes the update of the operation state information sent from the remote terminal. If it is determined that the object received in step S4210 is operation trajectory information, the process advances to step S4250 to additionally store the operation trajectory information in the electronic object storage unit 331, and the process ends. This process completes adding the operation trajectory information sent from the remote terminal.

<Panel surface image forming process>
A process for forming a board image will be described with reference to FIG. FIG. 13 is a flowchart of the board image forming process. This process is performed by the panel image forming unit 332, and a projection image and a background image are formed from the electronic object stored in the electronic object storage unit 331 by judging the instruction to generate the projection image for shooting from the writing image information generating unit 312. To do. There are two types of projection images. The first is an image for writing and shooting projected when shooting writing on the whiteboard at the site. This is for example a blank image. The second is a board sharing image for projecting a whiteboard writing image and operation information of the remote terminal by this terminal. These are collectively called a projection image.

  In step S4300, the set state of the projection instruction for the photographic image instructed from write image information generation unit 312 is determined. If the projection instruction for the photographic image is set in step S4310 (YES), the process for projecting the photographic image is performed from step S4320 to step 4324.

  On the other hand, if the projection instruction of the shooting image is not set in step S4310 (NO), the whiteboard writing image, the remote base operation state, and the projection image for knowing the operation locus are shared between the main site and the remote site, that is, the board surface sharing. A process for forming a background image and a background image is performed. This is performed from step S4330 to step S4352.

  First, the process of projecting an image for photographing will be described from step S4320 to step 4324. In step S4320, image data for projection is initialized. Specifically, the projection image is initialized with a white image that is a photographing image projected onto the whiteboard. In step S4322, the created shooting image is stored in the projection image storage unit 335 as a projection image. In step S4324, the image for shooting is stored in the background image storage unit 336 as a background image. With the above processing, the image for writing and shooting stored in the projection image storage unit 335 by the projection image output unit 333 is projected onto the whiteboard by the image projection module 212.

  Next, the process of forming the projection image for sharing the board and the background image will be described in steps S4330 to S4352. In step S4330, the update state of the electronic object stored in the electronic object storage unit 331 is checked. If there is no update (NO), it is not necessary to form a new projection image and a background image, and the process ends.

  On the other hand, if the electronic object is updated (YES), the process proceeds to step S4340 to form a new projection image and background image. In step S4340, the projection image is initialized. Specifically, initialization is performed with a white image as a background color projected on the whiteboard. Of the operation trajectory information stored in the electronic object storage unit 331 in step S4342, an operation in which the generation terminal 1103 is a terminal other than the terminal executing this processing (referred to as a terminal at another base). Draw trajectory information. Specifically, it corresponds to the operation trajectory information 403 shown in FIG. The drawing target is the confirmed operation trajectory information shown in FIG. In step S4342, a solid circle of a certain size is drawn centered on the position indicated by the hand position coordinates 1111 of this information. There can be a plurality of pieces of operation trajectory information. For this reason, as shown in FIG. 4D, a plurality of circles are drawn so as to be drawn as a locus of the writing operation.

  In step S4344, among the operation state information stored in the electronic object storage unit 331, operation state information is drawn such that the generation terminal 1003 is a terminal at another site. Specifically, as shown in FIG. 10 (d), the operation status information in which the presence / absence of person 1005 is “present”, the hand detection 1010 is “present”, and the operation type 1010 is “write” is shown in FIG. Rendered as a write-type operation state 403 as shown in (c). Further, as shown in FIGS. 10C and 10E, the operation state information in which the presence / absence of person 1005 is “present”, the hand detection 1010 is “present”, and the operation type 1010 is “point indication” is shown in FIG. It is rendered as a point indication type operation state 805 as shown in FIG. As shown in FIGS. 10 (c) and 10 (e), the operation state information in which the presence / absence of a person 1005 is “present” and the hand detection 1010 is “not present” is an operation of a non-hand detection type as shown in FIG. Rendered as state 806. In the point instruction type and the writing type, the image of the person area information is drawn at the coordinate position indicated by the person position as shown in FIGS. 9C and 9D, respectively. Furthermore, a circle of a certain size is drawn around the coordinates indicating the position of the hand. The hand non-detection type draws an image of person area information at the coordinate position indicated by the person position as shown in FIG.

  In step S4346, the generation terminal stored in the electronic object storage unit 331 draws a writing image at another site. A specific example will be described using the terminal 102 base shown in FIG. The terminal 102 projects a written image corresponding to “a + b =” generated by the terminal 101. The write image information in which the object type in FIG. 6E is the write image and the generation terminal information is the terminal 101 is the target. In step S4348, the formed image is stored in the projection image storage unit 335. This completes the formation of the board-sharing image for projection onto the whiteboard.

  Next, in step S4350, the generation terminal additionally draws the writing image at the main site with respect to the images generated up to step S4344. A specific example will be described using the terminal 102 base shown in FIG. As shown in FIG. 6E, the terminal 102 holds written image information obtained by writing the main site (terminal 102) as an image. Here, the writing image of writing “2” is acquired by photographing or the like, and additionally drawn on the board sharing image stored in step S4348. This completes the formation of the background image shown in FIG. In step S4352, the image formed in step S4350 is stored in the background image storage unit 336, and the process ends.

  As described above, in the electronic conference system using the conventional camera-equipped projector according to the present embodiment, it is possible to notify a remote place of operations such as writing and pointing on the whiteboard. By detecting the writing status on the whiteboard from the photographed image of the whiteboard and notifying the remote location of the information indicating the writing status, the remote location side can know the writing area on the whiteboard. In addition, by detecting the pointing instruction to the whiteboard from the photographed image of the whiteboard and notifying the remote area of the information on the area where the pointing instruction is made, the area pointing at the local site is shared with other bases become able to.

[Embodiment 2]
In the second embodiment, a method for distinguishing operation states depending on whether a face is detected in a person area will be described. The operation state is distinguished by template matching the hand shape of the first embodiment. In the case where a person explains to a conference participant while pointing to the board of the whiteboard, as shown in FIG. 9C, the face of the person generally pointing to the whiteboard is in a state of facing the front. . Here, the operation information generation process of this embodiment will be described with reference to FIG. FIG. 15 is a flowchart of processing for generating operation information based on a captured image according to the second embodiment. This processing is performed by the operation information generation processing unit 341, and operation information is generated based on the operation state of the person obtained by analyzing the captured image. The processing in steps S4100 to S4140 in FIG. 7 is omitted because it is executed in common in this embodiment. FIG. 15 shows processing after branching when the determination in step S4110 in FIG. 7 is YES. The difference from the processing of the first embodiment described in step S4150 and subsequent steps in FIG. 7 is that steps S4190 to S4191 are added as branch destinations in S4152 and their splits. There is no change in the operation information to be generated.

  In step S4150, a hand is detected for a new person area. Here, hand detection is performed by methods such as template matching and skin color detection. The shape is not judged. Although steps S4150 and S4152 in FIG. 15 are described differently from steps S4150 and S4152 in FIG. 7, they may be the same processing. If a hand is not detected in step S4152 (NO), the process proceeds to step S4154, and the process proceeds to a flow for generating undetected type operation state information. Since the processing is as described with reference to FIG. If a hand is detected in step S4152 (YES), the process proceeds to step S4190, and a face is detected for the person area.

  In step S4191, when a face facing front is detected (YES), the process proceeds to step S4156 to generate point indication type operation state information. On the other hand, in step S4191, if a face facing front is not detected (NO), the process proceeds to step S4158 to generate write type operation state information.

  As described above, it is possible to determine whether the operation state is a point instruction or writing based on the fact that the person's face is facing the front.

[Embodiment 3]
In the third embodiment, a method for switching operation states by voice will be described. In the first and second embodiments, the operation state is distinguished by detecting the shape of the hand and the face by image processing. In the case where a person explains to a conference participant while pointing to the board of the whiteboard, the words are explained to the conference participant as shown in FIG. 9C.

  The software configuration and operation information generation processing of this embodiment will be described with reference to FIGS. FIG. 16 is a diagram illustrating a software configuration according to the third embodiment. FIG. 16 is a software configuration diagram of the present embodiment, and the difference from the first and second embodiments shown in FIG. 3 is that an operation state switching unit 345 is provided. The operation state switching unit 345 notifies the operation information generation unit 341 whether the operation state is a point instruction or writing. The operation state is switched according to the input from the HID input unit 362 or the voice input unit 363. In the third embodiment, the operation state switching unit 345 recognizes and switches the voice input from the voice input unit 363. That is, it is determined by voice recognition that a word to be explained is given to participants in the conference shown in FIG. Specifically, the point state is switched during a period during which voice recognition is being performed, and the operation state is switched during a period during which voice recognition is not being performed.

  FIG. 17 is a flowchart of processing for generating operation information based on a captured image and a voice recognition state according to the third embodiment. This processing is performed by the operation information generation processing unit 341, and operation information is generated based on the operation state of the person obtained by analyzing the captured image and the operation state information determined by the operation state switching unit. The processing in steps S4100 to S4140 in FIG. 7 is omitted because it is executed in common in this embodiment. FIG. 17 shows processing after branching when the determination in step S4110 in FIG. 7 is YES. Differences between the processing of the first embodiment described with reference to FIGS. 17 and 7 are steps S4150 to S4152, and steps S4194 to S4195. There is no change in the generated operation information.

  In step S4150, a hand is detected for a new person area. Here, hand detection is performed by methods such as template matching and skin color detection. The shape is not judged. Although steps S4150 and S4152 in FIG. 15 are described differently from steps S4150 and S4152 in FIG. 7, they may be the same processing. If no hand has been detected in step S4152 (NO), the process proceeds to step S4154, and the process proceeds to a flow for generating undetected type operation state information. Since the processing is as described with reference to FIG. If a hand is detected in step S4152 (YES), the process proceeds to step S4194, and an operation state switching state by voice recognition by the operation state switching unit 345 is detected. The determination in step S4194 is, for example, whether or not there is an utterance by the operator. If there is an utterance, it is determined as a point instruction type, and if not, it is determined as a writing type.

  If it is determined in step S4195 that the operation state is a point instruction (YES), the flow advances to step S4156 to generate point instruction type operation state information. On the other hand, if it is determined in step S4191 that it is not a point instruction type (NO), the flow advances to step S4158 to generate write type operation state information.

  As described above, it is determined by voice recognition whether a person is uttering voice for explanation, and a point instruction is given during a period of voice recognition, and an operation state is shown as being in a writing state during a period of no voice recognition. It becomes possible to switch.

[Embodiment 4]
In the fourth embodiment, a method for manually switching the operation state will be described.
This is a method in which a person clearly switches the operation state using a HID device such as a mouse or a keyboard. As shown in the software configuration diagram of FIG. 16, in the present embodiment, the operation state switching unit 345 switches whether the operation state is a point instruction or writing as in the third embodiment. The operation state is switched by the HID input unit 362. Specifically, point indication and writing are switched by mouse or keyboard input.

  FIG. 18 is a flowchart of a process for generating operation information based on the captured image and manual operation state switching information in the fourth embodiment. This processing is performed by the operation information generation processing unit 341, and operation information is generated based on the operation state of the person obtained by analyzing the captured image and the operation state information determined by the operation state switching unit. The processing in steps S4100 to S4140 in FIG. 7 is omitted because it is executed in common in this embodiment. FIG. 18 shows processing after branching when the determination in step S4110 in FIG. 7 is YES. Differences between the processing of the first embodiment described with reference to FIGS. 18 and 7 are steps S4150 to S4152, and steps S4192 to S4193. There is no change in the generated operation information.

  In step S4150, a hand is detected for a new person area. Here, hand detection is performed by methods such as template matching and skin color detection. The shape is not judged. Although steps S4150 and S4152 in FIG. 15 are described differently from steps S4150 and S4152 in FIG. 7, they may be the same processing. If no hand has been detected in step S4152 (NO), the process proceeds to step S4154, and the process proceeds to a flow for generating undetected type operation state information. Since the processing is as described with reference to FIG. If a hand is detected in step S4152 (YES), the process proceeds to step S4194, and an operation state switching state manually determined by the HID input unit by the operation state switching unit 345 is detected. Switching by HID is performed by actively instructing whether the current operation (or an operation to be performed in the future) is writing or pointing, for example, using a pointing device or a keyboard.

  If it is determined in step S4195 that the operation state is a point instruction (YES), the flow advances to step S4156 to generate point instruction type operation state information. On the other hand, if it is determined in step S4191 that it is not a point instruction type (NO), the flow advances to step S4158 to generate write type operation state information. As described above, the operation state can be manually switched.

[Other Embodiments]
Moreover, this embodiment is implement | achieved also by performing the following processes. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (10)

A conference terminal device,
Photographing means for photographing the writing object;
Projecting means for projecting an image onto the writing object;
Control means for controlling the conference terminal device;
Communication means for communicating with other conference terminal devices,
The control means includes
Detecting movement of a person's hand from the image taken by the photographing means, and sending operation information indicating an image object corresponding to the detected hand movement to the other conference terminal device by the communication means;
A conference terminal device, wherein an image object is generated based on the operation information received by the communication unit from the other conference terminal device, and is projected by the projection unit. The control means further detects the shape of the hand along with the movement of the person's hand, and if the shape is determined to indicate writing on the writing object, the operation information is used as the operation information. Sending trajectory information indicating the trajectory of movement to the other conference terminal device;
When the trajectory information is received from the other conference terminal device by the communication unit, an image object indicating a trajectory range is generated based on the trajectory information and projected by the projection unit. Item 2. The conference terminal device according to Item 1. The control means further detects the shape of the hand together with the movement of the person's hand, and if the shape is determined to be pointing pointing to the object to be written, the operation information includes the position of the person's hand. Operation information to be sent to the other conference terminal device,
When the operation information is received from the other conference terminal device by the communication unit, an image object indicating the position of the hand is generated based on the operation information and projected by the projection unit. The conference terminal device according to claim 1. When the control unit detects a person from the image captured by the imaging unit and does not detect the hand, the control unit displays operation information indicating an image object corresponding to the detected person by the communication unit. To the terminal device,
The image object corresponding to a person is generated based on the operation information and projected by the projection unit when the operation information is received from the other conference terminal device by the communication unit. The conference terminal device according to 1.   The control means detects a human face from an image photographed by the photographing means, and based on the detected face direction, the shape of the hand indicates writing to the writing object, or to the writing object. 4. The conference terminal device according to claim 2, wherein it is determined whether or not pointing is indicated. Further comprising means for detecting speech;
The control means determines whether the shape of the hand indicates writing on the writing object or pointing on the writing object based on the detected voice. Or the conference terminal device of 3. A user interface means for receiving an input by an operator;
3. The control unit according to claim 2, wherein the control unit determines, based on the input, whether the shape of the hand indicates writing on the writing object or pointing on the writing object. 3. The conference terminal device according to 3. The control means detects a written image written on the object to be written from images taken by the photographing means, and sends written image information indicating the detected written image to the other conference terminal device by the communication means. And
8. When receiving the written image information from the other conference terminal device, an image object is generated based on the written image information and projected by the projecting unit. The conference terminal device according to one item.   An electronic conference system, wherein a plurality of the conference terminal devices according to claim 1 are connected by a communication network. By a computer having a photographing means for photographing a writing object, a projecting means for projecting an image on the writing object, and a communication means,
Detecting the movement of a person's hand from the image photographed by the photographing means and transmitting operation information indicating an image object corresponding to the detected hand movement to the other conference terminal device by the communication means;
A program for generating an image object based on the operation information received by the communication unit from the other conference terminal device and causing the projection unit to project the image object.

Images