JP2015127925A - Information processing device, method of controlling information processing device, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing device that improves operability by specifying the type of an electronic object and switching a sharing method when sharing an information processing device other than itself and the electronic object.SOLUTION: An information processing device extracts a physical object as an electronic component corresponding to the physical object from image information obtained by photographing a projection plane as a photographic plane, receives an electronic object from an information processing device other than itself, and projects a projection image generated based upon the electronic object on the projection plane. The information processing device determines which of a synchronous type and a permanent type the electronic object is based upon features of the physical object, and if the physical object has change, the information processing device transmits information associated with the electronic object to the information processing device other than itself when the electronic object is the synchronous type, but does not transmit the information associated with the electronic object when the electronic object is the permanent type.

Description

  The present invention relates to an information processing apparatus, an information processing system, and a computer program for sharing image information captured by a camera of each terminal among a plurality of bases via a network and displaying the information on each terminal.

  A projection apparatus typified by a liquid crystal projector is provided with an imaging means such as a camera. For example, the projection apparatus disclosed in Patent Document 1 detects the timing when paper is placed on the projection plane / imaging plane, captures the projection plane in accordance with the timing, and acquires the data as data.

  In addition, an electronic conference system is also provided in which information is shared and discussed at a plurality of locations via a network. For example, when a terminal device disclosed in Patent Document 2 inputs data to a shared whiteboard, a public mode for writing information that all the participants want to share and a private mode for writing information that is not to be shared with other bases Both.

  A tangible interface is also provided as an interface for associating information with a real object and allowing the user to manipulate information without shape. For example, Non-Patent Document 1 discloses a system that can share in real time what is handwritten at its own site. Non-Patent Document 2 discloses a system that associates an actual object with information and performs operations such as an information display method and composition in accordance with physical operations (movement and superposition) of the object. . Further, Patent Document 3 discloses a command that allows an image displayed on a projection plane / photographing plane to be operated by a simple operation in which an object having a predetermined shape is placed on the projection plane / photographing plane and a person moves the object. Proposed input method.

JP 2011-215441 A JP-A-10-289175 JP 2007-11276 A

MIT Media Lab Tangible Media Group / Clear Board <http: // tangible. media. mit. edu / project / clearboard /> (confirmed on May 7, 2013) MIT Media Lab Tangible Media Group / Sense Board <http: // tangible. media. mit. edu / project / senseboard /> (confirmed on May 7, 2013)

  As described above, an electronic conference system in which both a physical object and an electronic object are displayed using a projector with a camera is assumed. In such a system, the conference terminal at the local site takes a physical object (book, catalog, whiteboard writing) and converts it to an electronic object (image data or electronic document data). The conference terminal transmits data to the conference terminal on the other site side to share information. At this time, two use cases related to information sharing can be roughly divided.

  There may be a first use case in which information written on a sticky note or whiteboard at a local site is desired to be shared with a conference participant at another site. At this time, in order to establish communication between the conference participant at the local site and the conference participant at the other site, it is necessary to synchronize the sites even when the information changes. If the physical participant moves the position of the physical object on the imaging plane, the conference terminal at the other site wants to change the display position of the electronic object on the projection plane. Here, for example, sticky notes physically attached to the whiteboard and information written on the whiteboard are referred to as physical objects. If the physical object is excluded from the imaging plane by the conference participant at the local site, the conference terminal at the other site wants the electronic object to be excluded from the imaging plane. As described above, when the physical object changes, there is a first request that the electronic object displayed on the conference terminal at the other base should be synchronized with the change.

  Here, in the conference system that satisfies the first demand, when a projection plane and a photographing plane are provided perpendicular to the ground, the physical object that can be photographed is limited to a relatively lightweight object. Alternatively, it is possible to photograph and print physical objects in several directions in advance, but there is a problem of increasing the time and labor for holding a conference. If a projection plane / photographing plane is provided horizontally with respect to the ground, a heavy physical object can be photographed. However, it is difficult to see the same information at the same time because the conference is held in such a way that multiple people look into it from various angles.

  There may be a second use case where a book, catalog, or three-dimensional object at hand is desired to be shared with a conference participant at another location. Considering readability, when a plurality of people participate in the conference, it is desirable that the whiteboard (projection plane / photographing plane) be arranged perpendicular to the ground. However, it is difficult for participants to keep fixing heavy books, catalogs, and three-dimensional objects on the shooting surface perpendicular to the ground. Therefore, if a physical object is photographed on the local site side, the conference terminal should be displayed as an electronic object on both the local site side and the other site side even if it is excluded from the photographing surface. As described above, even if the physical object changes, there is a second request that the electronic object displayed on the conference terminal at the local site / other site does not want to be synchronized with the change.

In the conference system that satisfies the second demand, at the timing when the position of the physical object is operated at the local site, a difference occurs in the object viewed from the electronic object displayed at the other site. By digitizing all materials, information can be shared among multiple sites. However, in order to operate an electronic object, it is necessary to use an electronic pen or gesture input, and there is a problem in operability. When the electronic pen is used, new hardware such as an ultrasonic sensor is required separately, which increases the product cost. Gesture input can use the original camera, so no new hardware is required, but there are still problems with recognition accuracy.
An object of the present invention is to provide an information processing device that improves operability by specifying a type of the electronic object and switching a sharing method when the electronic object is shared with an information processing device other than the device itself. .

  An information processing apparatus according to an embodiment of the present invention shares captured image information among a plurality of information processing apparatuses via a network. The information processing apparatus extracts a physical object from image information captured using a projection plane as an imaging plane, generates an electronic object corresponding to the physical object, and generates an electronic object from an information processing apparatus other than the own apparatus. Receiving means for receiving, projecting means for projecting a projection image generated based on the generated or received electronic object onto the projection plane, and the generated based on characteristics of an image area corresponding to the physical object The electronic object type is a synchronous type drawn on a projection image projected by an information processing device other than the own device in synchronization with the change of the physical object on the photographing surface of the own device, or information other than the own device without being synchronized Judgment means for judging whether or not to permanently project onto the projection surface of the processing apparatus, and the synchronization type Others and transmission means for transmitting the electronic objects that are determined to persistent type information processing apparatus other than the self apparatus. When the change occurs in the physical object, the transmission means transmits information about the electronic object to an information processing apparatus other than the own apparatus when the electronic object is the synchronization type, and the electronic object When it is the permanent type, information regarding the electronic object is not transmitted.

  According to the information processing apparatus of the present invention, when an electronic object is shared with an information processing apparatus other than its own apparatus, the operability can be improved by specifying the type of the electronic object and switching the sharing method. . Therefore, when a plurality of information processing apparatuses that display physical objects and electronic objects on the screen share the electronic objects, the projection display by each apparatus is appropriately controlled without complicated operations.

It is a network structural example of the conference system of this invention. It is a hardware structural example of a conference terminal. It is an example of a software configuration of a conference terminal. It is a flowchart of the process which produces | generates an electronic object from a picked-up image. It is an example of the electronic object produced | generated from the picked-up image. It is a flowchart of the process which determines an object type from a still image. It is an example of the object type determined based on the still image characteristic. It is a flowchart of the process which determines an object type from a moving image. It is an example of the object type determined based on animation characteristics. It is a flowchart of the process which performs update implementation determination. It is a flowchart of the process which forms a projection image from an electronic object. It is an example of the electronic object which the electronic object storage part has memorized. It is an example which forms a projection image from an electronic object. It is an example of the user interface which designates an object type. It is a flowchart which designates an object type according to a position. This is an example of specifying an object type according to a position. It is a flowchart of the process which changes an object type by putting a physical object on displaying an electronic object. It is an example of the imaging | photography surface by which the object type was changed as a result of the process of FIG. It is an example of the object type determination process to which distance information is applied.

Example 1
FIG. 1 shows a network configuration example of the conference system of the present invention. The conference terminals 101 and 102 are connected to the Internet 105 via gateway servers 103 and 104, respectively. The conference terminal is an information processing apparatus that can transmit and receive data to and share data with each other via the gateway servers 103 and 104 and the Internet 105. In the following description, attention is paid to the processing performed on the conference terminal 101 side. Here, it is assumed that the conference terminal 101 side is treated as “present terminal” and the conference terminal 102 side is treated as “opposite terminal”. It should be noted that this relationship is switched when attention is paid to the conference terminal 102 side. Further, in the present invention, the number of opposing terminals is not limited to one, but it will be clearly stated in advance that the present invention can be applied even when a plurality of terminals are configured.

  FIG. 2 shows a hardware configuration example of a conference terminal that functions as the information processing apparatus of the present invention. Since the conference terminal 101 and the conference terminal 102 have the same configuration, the conference terminal 101 will be described below as an example. In this embodiment, a projector with a camera is used as a conference terminal. Although a projector with a camera is described here as a dedicated hardware device constituting the conference system, a conference terminal can also be configured by combining a projector and a smartphone with a camera. It is also possible to configure a conference terminal by connecting a commercially available Web camera to the projector. A camera-equipped smartphone with a projector can also be used as a conference terminal.

  The conference terminal 101 includes a CPU 201, a RAM 202, a ROM 203, an external device controller 204, and an internal device controller 205. The CPU 201 to the internal device controller 205 are connected via the main bus 200. In addition, an image capturing module 211, an image projecting module 212, an HID module 213, and an audio input module 214 are connected to the external device controller 204. An internal storage device 221 and a network module 222 are connected to the internal device controller 205. The CPU 201 can communicate with these control modules via the external device controller 204 and the internal device controller 205.

  CPU is an abbreviation for Central Processing Unit. The CPU 201 is a unit for executing a program 300 described later. RAM is an abbreviation for Random Access Memory. The RAM 202 is a readable / writable memory for data necessary for the CPU 201 to execute the program 300. ROM is an abbreviation for Read Only Memory. A ROM 203 is a read-only memory for reading data necessary for the CPU 201 to execute the program 300.

  The external device controller 204 is a module for controlling external devices such as the image capturing module 211 and the image projection module 212. The image photographing module 211 is a module for photographing an image on the photographing surface as a photographed image. The image projection module 212 is a module for displaying the projection image generated by the CPU 201 on the projection plane. An HID (Human Interface Device) module 213 is a module for receiving an operation from a human.

  The voice input module 214 is a module for receiving voice input from a human. For example, a CCD camera module can be used as the image capturing module 211, and a liquid crystal panel projection module can be used as the image projection module 212. A mouse can be used as the HID module 213 and a microphone can be used as the voice input module 213.

  The internal device controller 205 is a module for controlling internal devices of the conference terminal 101, and is a module for controlling the internal storage device 221 and the network module 222. The internal storage device 221 is a secondary storage device for storing data. The network module 222 is a module for transmitting / receiving information to / from another device on the network. As the network module 222, not only a network module supporting a wired LAN but also a network module supporting a wireless LAN can be applied. That is, the terminal and the opposite terminal transmit / receive data to / from each other via the network module 222.

  FIG. 3 shows a software configuration example of the conference terminal. When the conference terminal of the present invention realizes the conference system of each embodiment, the CPU 201 reads and executes the program 300. The program 300 is stored in the RAM 202 or ROM 203. The connection information 301 is setting data that stores connection information such as the IP address of the opposite terminal that is the connection destination.

  The captured image input unit 311 receives a captured image captured by the image capturing module 211. In this embodiment, the photographed image input unit 311 performs preprocessing for geometric correction and color adjustment. The electronic object generation unit 312 analyzes the captured image and detects a partial region where the physical object exists. Then, a first electronic object is generated based on the partial area.

  The electronic object type determination unit 351 obtains a partial area that is a source of the first electronic object and a still-image characteristic or a moving-image characteristic that changes in time series at the shooting time of a peripheral area that includes the partial area. . Then, the electronic object type determination unit 351 determines the object type of the first electronic object based on the characteristics of the partial area or the peripheral area.

  Based on the object type of the first electronic object, the update execution determining unit 352 determines whether information regarding the first electronic object should be registered / updated in the terminal and the opposite terminal. The local electronic object storage unit 313 stores, in the electronic object storage unit 331, information on the electronic object determined to be updated in the terminal by the update execution determination unit 352.

  Based on the connection setting 301, the electronic object packet transmission / reception unit 314 is configured to transmit the electronic object packet in a format that can be transmitted / received on the network based on the information of the electronic object that is determined to be updated by the update execution determination unit 352. Is generated. Then, the data is transmitted to the opposite terminal via the network module 222.

  Based on the connection setting 301, the electronic object packet receiving unit 321 receives the electronic object packet transmitted from the opposite terminal, and extracts the second electronic object included in the electronic object packet. The remote electronic object storage unit 322 stores the second electronic object extracted by the electronic object packet reception unit 321 in the electronic object storage unit 331.

  Here, the electronic object packet transmitted and received by the electronic object packet transmitting unit 314 and the electronic object packet receiving unit 321 includes position information, image information, and the like on the projection plane of each electronic object. As a data format of the electronic object packet, various formats such as a JSON (Java Script (registered trademark) Object Notation) format and an XML (Extensible Markup Language) format can be used.

  The projection image forming unit 332 forms a projection image from the first and second electronic objects stored in the electronic object storage unit 331. The projection image output means 333 projects the projection image onto the projection plane using the image projection module 313.

  The gesture input unit 361 is a unit for analyzing the gestures / hand gestures of the conference participants from the captured images captured by the image capturing module 211 and notifying the electronic object operation unit 364. The HID input unit 362 is a unit for receiving an operation performed by the HID module 213 and notifying the electronic object operation unit 364. The voice input unit 363 is a unit for analyzing an operation performed by the voice of the conference participant from the voice data input from the voice input module 214 and notifying the electronic object operation unit 364. The electronic object operation unit 364 changes the electronic object stored in the electronic object storage unit 331 based on the operation notified from the other module.

  Here, the program 300 may have other general functions required for the electronic conference system. For example, in order to designate the opposite terminal, a connection destination setting dialog box for setting a connection destination conference terminal can be provided. In order to select the opposite terminal, it is possible to provide a function of inquiring about a terminal that can be connected to a server existing outside via the network module 222 and a function of notifying the dialog box as a candidate. Further, the program 300 can have a function of encrypting an electronic object packet transmitted by the electronic object packet transmission means 314 in order to ensure security in the conference. The program 300 can have a function of decoding the electronic object packet received by the electronic object packet receiving unit 321. Since these are general functions used in the electronic conference system of the prior art and do not directly affect the essence of the present invention, details are omitted.

  Hereinafter, the features of the present invention will be described focusing on the electronic object generation unit 312, the electronic object type determination unit 351, the update execution determination unit 352, and the projection image formation unit 332 that are executed by the CPU 201 as hardware.

<Generate electronic objects>
FIG. 4 is a flowchart of processing for generating an electronic object from a captured image, which is performed by the electronic object generating unit 312. FIG. 5 shows an example in which the electronic object generation unit 312 generates an electronic object from a captured image. This processing is performed for each frame of the moving image sent from the photographed image input means 311.

  The electronic object generation means 312 determines whether it is suitable for the shooting timing from the captured image (S400). A conference participant can perform an operation via the gesture input unit 361, the HID input unit 362, and the voice input unit 363, and can receive and determine the shooting timing from the electronic object operation unit 364. For example, the gesture input unit 361 analyzes a photographed image photographed by the image photographing module 211, and can determine that it is a photographing timing with a specific gesture / hand gesture performed therein. The gesture input unit 361 can also determine that it is the shooting timing when the moving object disappears from the captured image. Clicking with the mouse as the HID module 213 is interpreted by the HID input means 302, and it can be determined that it is the photographing timing. The voice input unit 263 interprets the voice input by the voice input module 213, and the timing at which the specific keyword is detected can be determined as the shooting timing.

  The electronic object generation unit 312 branches the process based on the result of determining whether it is the above photographing timing (S402). If it is not the shooting timing, the processing for this frame ends. If it is the shooting timing, the electronic object generation unit 312 acquires the shooting time (S403). This is a generation time given during the electronic object generation process described later.

  The electronic object generation unit 312 performs background difference processing on the captured image to separate the captured image from the background portion of the whiteboard and the foreground portion including the physical object in units of pixels (S404). At this time, preprocessing for detecting the contours and colors of the hands and arms in the photographed image and estimating and complementing the portions hidden by them may be further added. An example of this processing is shown in FIG. The black pixel portion means the foreground portion, and the white pixel portion means the background portion.

  The electronic object generation unit 312 extracts a plurality of contours for the pixels in the foreground portion, and sets the plurality of contours as partial regions (S406). At this time, it is desirable to perform preprocessing so that adjacent foreground portions are joined together. In the processing example shown in FIG. 5B, five partial areas 501 to 505 are extracted.

  The electronic object generation unit 312 performs the following processing on each partial area. First, the electronic object generation unit 312 selects a partial area to be processed (S408). The electronic object generation means 312 calculates the area of the partial area to be selected, and if it is less than a certain value, selects another partial area (S410). This is necessary to exclude a partial region generated from an erroneously detected isolated point. Since the partial area is a pass point sequence, the area can be calculated using a brushing method or the like. Alternatively, the area of the bounding box that includes the partial region can be used as an approximate value of the area of the partial region itself.

  The electronic object generation unit 312 generates an electronic object based on the information of the partial region having a certain area or more (S412). The generated electronic object has attributes of electronic object ID, physical object presence / absence, image information, position information, object type information, generation terminal information, stationary start time, and stationary time (FIG. 5C). The electronic object ID is an identifier that is sequentially assigned to each electronic object. The presence / absence of a physical object is a flag indicating whether a physical object is present on the base-side projection plane where the terminal that generated the electronic object is present. The image information is an image obtained by cutting out a portion indicated by the partial area on the captured image. The position information is the coordinates of the upper left origin of the partial area.

  The object type information is identification information for determining a behavior when a physical object moves, for example, whether to perform an update process. The object type includes a deleted type, an undetermined type, a permanent display type, and a synchronous display type. In the present embodiment, the permanent display type refers to an electronic object that is permanently projected onto a projection plane of an information processing apparatus other than its own apparatus without being synchronized. For example, it is an electronic object that is displayed at a fixed position on the projection plane of the apparatus other than the own apparatus together with the projection plane of the own apparatus even if the physical object is removed and moved from the imaging plane captured by the own apparatus. The synchronous display type refers to an electronic object drawn on a projection image projected by an information processing apparatus other than the own apparatus in synchronization with a change in a physical object on the photographing surface of the own apparatus. For example, if the physical object moves, the electronic object on the opposite terminal side is also an electronic object that undergoes the same movement. In the synchronous display type, when the physical object is removed on the terminal side, the electronic object on the opposite terminal side is not displayed. The undetermined type is given to an electronic object for which an electronic object corresponding to a physical object has been photographed but cannot be determined as either a synchronous display type or a permanent display type. The deleted type is given to the permanent display type electronic object when there is an explicit removal designation from the conference participant. The initial value of the object type when the electronic object is generated is an undetermined type. The synchronous display type and the permanent display type may be simply referred to as a synchronous type and a permanent type, respectively.

  The generated terminal information uses an identifier representing the terminal that generated the electronic object. In this embodiment, it is assumed that this terminal (that is, terminal 101). The still start time is the time when the physical object that is the source of the electronic object starts to stop in the captured image, and is the previously determined shooting time. The stationary time is the time at which the physical object that is the source of the electronic object remains stationary in the captured image. This is calculated in a later process.

  Hereinafter, in order to simplify the description, an electronic object having the electronic object ID 511 is referred to as an electronic object 511. In the example of FIG. 5C, electronic objects 511 to 514 corresponding to the partial areas 501 to 504 are generated. Since the partial area 505 has a small area, an electronic object is not generated. The electronic object generation means 312 determines whether there is another partial area, and if it still remains, repeats these processes (S414). Through the above processing, an electronic object is generated based on the partial area in the captured image including the physical object.

  Next, the electronic object generation unit 312 detects whether or not the electronic object held in the electronic object storage unit 331 is excluded. First, a target electronic object is selected from the electronic object storage unit 331 (S450). If the generation terminal of the electronic object is not the terminal, the electronic object generation unit 312 ends the process for the electronic object (S452). If there is a partial area corresponding to the electronic object, the electronic object generation unit 312 ends the process for the electronic object (S454). If the generation terminal of the electronic object is the terminal and there is nothing corresponding to the partial area, the electronic object generation unit 312 generates a new electronic object without the physical presence or absence of the electronic object ( S456). The electronic object generation unit 312 determines whether there is another electronic object in the electronic object storage unit 331, and repeats these processes again if it remains (S458). Through the above processing, the electronic object generated based on the physical object excluded from the captured image is generated as a new electronic object with or without physical presence.

  Next, the electronic object type determining means 351 determines the object type of each electronic object as either a still image characteristic at the shooting time or a moving image characteristic focusing on time-series changes in the partial area and its peripheral area. Decide on the basis. It should be noted that various determination methods can be applied simultaneously by combining a plurality of object type determination means.

  In addition, only a brief description will be given here, but if it is specified to delete an electronic object of the permanent display type, it is only necessary to specify the deleted object type and add it to the generated electronic object. . As a method for selecting an electronic object to be deleted, for example, as in Non-Patent Document 2, a method can be used in which a physical object on which a deletion marker is written is superimposed and photographed. Alternatively, the position of the electronic object may be moved via the HID input unit 362, and it may be determined that deletion has been selected when dragging over the icon or dragging outside the display screen. Alternatively, the voice input unit 363 may issue a deletion instruction while the gesture input unit 361 or the HID input unit 362 specifies an electronic object.

<Determining object type based on still image features>
Consider a feature on an image obtained from a still image when a physical object to be captured as a synchronous display type and a physical object to be captured as a permanent display type are mixed.

  Consider a use case where a physical object to be captured as a synchronous display type is fixed on a photographing surface by a magnet. A physical object to be captured as a permanent display type cannot be fixed on the photographing surface with a magnet, but is fixed on the projection surface by holding it by hand. That is, by detecting a magnet or a hand from the captured image, the object type to be given to the electronic object can be determined.

  As a method for determining the object type, another use case using a paper sticky note in which a paste having a weak adhesive force is attached to the back surface is also conceivable. It is possible to determine whether or not the physical object photographed in the partial area is a tag by printing an identification marker for identifying the tag in advance or limiting the colors of the tags that can be used. When it is determined that the physical object is a tag, it can be determined that the object type of the electronic object is a synchronous display type.

  FIG. 6 is a flowchart for determining the object type of the electronic object based on the characteristics of the image area corresponding to the physical object in the still image. FIG. 7 is an example of a physical object that determines an object type of an electronic object based on still image characteristics. First, the electronic object type determination unit 351 determines whether or not the target electronic object is physical existence = there is no physical object, and does not perform the following processing if there is no physical object (S600). The electronic object type determination unit 351 extracts a partial image on the projection image corresponding to the position / information of the electronic object (S601). The electronic object type determination unit 351 determines whether or not the hand is superimposed and adjacent to the partial image and its peripheral part (S604). For example, it is determined that the hand 702 is superimposed on the physical object 701 illustrated in FIG. When the hands are superimposed / adjacent, the electronic object type determination unit 351 determines the object type of the electronic object as a permanent display type (S606).

  The electronic object type determination unit 351 determines whether an identification marker or a magnet is included in the partial image when the hands are not overlapped (S612). For example, identification markers 722 and 723 exist in the partial image obtained by photographing the physical object 721. Identification markers 732 and 733 exist in the partial image obtained by photographing the physical object 731. Arbitrary shapes can be used for these identification markers, and some information can be further given thereto.

  Further, magnets 742 and 743 exist in the partial image obtained by photographing the physical object 741. A magnet 752 exists in the partial image obtained by photographing the physical object 751. The presence of these can be determined by determining whether an image corresponding to a magnet is included in the image. An arbitrary shape or color used as a magnet may be specified in advance on the system side. An identification marker can be attached to the magnet itself. When an identification marker or a magnet is present in the partial image, the electronic object type determination unit 351 determines the object type of the electronic object as a synchronous display type (S614).

  If there is no identification marker or magnet in the partial image, the electronic object type determining means 351 specifies the background color in the partial area (S618). For example, each color included in the partial image is clustered, and the most frequently occurring color cluster can be estimated as the background color. At this time, it is desirable to exclude a color cluster often used for oil-based markers or the like when writing on a sticky note from the candidates for the background color. Next, it is determined whether the background color matches any one of the sticky note specific colors (S620). In many cases, commercially available paper sticky notes are lightly colored so that the writing can be clearly seen, and the colors of the respective sticky notes are easily distinguishable. By determining whether the background color matches the specific color of the tag, it is determined whether the physical object photographed in the partial area is a tag. The specific color of the sticky note may be determined in advance, or may be determined based on conditions such as lightness and saturation. Furthermore, in this embodiment, the determination is made based on the color, but the determination is based on the shape characteristics such as whether the width / height ratio of the partial image matches the width / height ratio of the tag. May be.

  When the background color matches the tag specific color, the electronic object type determination unit 351 determines the synchronous display type (S622). If they do not match, it is determined as an undetermined type (S624). If there is another electronic object to be processed, the process is repeated, otherwise the process ends (S630). Here, whether or not a specific image such as a color in the partial region, an identification marker, or a magnet is included is used as a material for determining the object type. In addition, the number of colors in the partial area, the average / distribution of the line thickness, and the like can also be used for determining the object type.

<Determining object type based on animated features>
Next, when the physical object to be captured as the synchronous display type and the physical object to be captured as the permanent display type are mixed, the feature on the image obtained from the moving image will be considered.

  Consider a use case where a physical object to be captured as a synchronous display type is fixed on a photographing surface by a magnet. A physical object to be captured as a permanent display type cannot be fixed on the imaging surface with a magnet, but instead is fixed on the projection surface by holding it by hand. At this time, the physical object to be captured as the synchronous display type can be fixed on the photographing surface for a long time. However, since the physical object to be captured as the permanent display type is manually fixed on the photographing surface by the conference participant, it is ideal to make it as short as possible considering the convenience of the conference participant. When this is used, first, all electronic objects (to be imported) should be provisionally determined as permanent display types, and a method of re-determining synchronous display types for electronic objects that have been stationary on the projection surface for a certain time or longer should be adopted. Can do. These behaviors are the same for use cases using paper sticky notes.

  FIG. 8 is a flowchart for determining the object type of the electronic object based on the characteristics of the image area corresponding to the physical object in the moving image. FIG. 9 is an example in which the object type of an electronic object is determined based on moving image characteristics. First, the electronic object type determination unit 351 determines whether the target electronic object is physical existence = not present, and does not perform the following processing when there is no physical object (S800). The electronic object type determination unit 351 compares the electronic object with the electronic object stored in the electronic object storage unit 331, and determines whether or not a new registration has been made (S802). If it is a new registration, the object type is set as an undetermined type (S804).

  Next, the electronic object type determination unit 351 compares the information stored in the electronic object storage unit 331 to determine whether the electronic object has moved since the previous shooting (S806). This is equivalent to determining whether the physical object that is the source of the electronic object is moving in the captured image. Actually, it is expected that the photographed image will be blurred due to the vibration of the desk where the terminal is installed during photographing. In these movement determinations, it is assumed that some variation is allowed with a certain threshold value. If it is determined that the electronic object is moving, the electronic object type determination unit 351 determines that the object type of the electronic object is an undetermined type (S808).

  If it is determined that the electronic object has not moved, the electronic object type determination unit 351 calculates the time during which the electronic object is stationary (S810). The electronic object type determination unit 351 determines whether the stationary time is equal to or less than the capture determination threshold that is an arbitrary threshold, and determines that the object type of the electronic object is an undetermined type if it is equal to or less than the capture determination threshold ( S814). If the stationary time is greater than the capture determination threshold, the electronic object type determination unit 351 determines whether the stationary time is less than or equal to the synchronization determination threshold. If the stationary time is less than or equal to the synchronization determination threshold, the object type of the electronic object is permanently displayed. The type is determined (S824). If the stationary time is larger than the synchronization determination threshold, the electronic object type determination unit 351 determines the object type of the electronic object as a synchronous display type (S826). If there is another electronic object to be processed, the process is repeated, otherwise the process ends (S830).

  Hereinafter, a specific behavior will be supplemented with reference to FIG. In this example, as the predetermined time, the capture determination threshold is 4 seconds and the synchronization determination threshold is 8 seconds. Also, it is assumed that no electronic object is registered in the electronic object storage unit 331.

  FIG. 9A is a photographed image taken on May 20, 2013, 9: 19: 42.250. A physical object 951 written “Mascot proposal!” Is photographed. At this time, no electronic object exists in the electronic object storage unit 331. Therefore, the electronic object type determination unit 351 determines that the object type of the electronic object 901 generated from the partial area obtained by photographing the physical object 951 is an undetermined type. Then, the electronic object type determination unit 351 stores the electronic object 901 in the electronic object storage unit 331 in a later process.

  FIG. 9B is a photographed image taken on May 20, 2013, 9: 19: 47.250. The physical images 961, 962, and 964 written as “mascot plan!”, “Inu”, and “downwardly slanting arrow”, and the physical object 963 of the “dog image” on hand are included in the photographed image. At this time, an electronic object 901 is stored in the electronic object storage unit 331. The electronic object type determination unit 351 compares the electronic object 911 and the electronic object 901 and determines that it has not moved. At this time, it is calculated from the difference between the stationary start time and the photographing time that the stationary time is 5 seconds. This exceeds the capture determination threshold and is less than the synchronization determination threshold. Therefore, the electronic object type determination unit 351 determines that the object type of the electronic object 911 is a permanent display type. The electronic object type determination unit 351 determines that the electronic objects 912, 913, and 914 are of an undetermined type because no electronic object corresponding to the electronic object storage unit 331 is stored.

  FIG. 9C is a photographed image taken on May 20, 2013, 9: 19: 52.250. The physical images 971, 972, and 974 written as “mascot plan!”, “Inu”, and “downwardly pointing arrow” are included in the photographed image. A physical object 973 of “dog” held by hand is excluded from the photographing surface. At this time, electronic objects 901, 912, 913, and 914 are stored in the electronic object storage unit 331. The electronic object type determination unit 351 compares the electronic object 921 and the electronic object 901 and determines that it has not moved. At this time, the stationary time is calculated as 10 seconds from the difference between the stationary start time and the photographing time. This stationary time exceeds the capture determination threshold and also exceeds the synchronization determination threshold. Therefore, the electronic object type determination unit 351 determines that the object type of the electronic object 921 is a synchronous display type.

  Since the physical object 973 is excluded from the imaging surface, the object type is not determined because the physical object 973 is taken in based on the presence or absence of physicality based on the electronic object 913. Therefore, the electronic object type determination unit 351 determines that the electronic object 923 is an undetermined type. If the same determination is made for the electronic objects 922 and 924, the stationary time is 5 seconds, and therefore, it is determined that the electronic objects 922 and 924 are of the permanent display type. That is, in this example, when the stationary time of the electronic object is 4 seconds or longer and within 8 seconds, the electronic object determining means determines that the electronic object is a permanent display type.

  FIG. 9D is a photographed image taken on May 20, 2013, 9: 12: 57.350. The physical images 981, 982, and 983 written as “mascot plan!”, “Inu”, and “downwardly pointing arrow” are included in the photographed image. At this time, electronic objects 901, 912, 913, and 914 are stored in the electronic object storage unit 331. The electronic object type determination unit 351 compares the electronic object 931 and the electronic object 901 and determines that they are not moved. At this time, the stationary time is calculated as 15 seconds from the difference between the stationary start time and the photographing time. This stationary time exceeds the capture determination threshold and also exceeds the synchronization determination threshold. Therefore, the object type of the electronic object 931 is a synchronous display type. If the same determination is made for the electronic objects 932 and 933, the stationary time is 10 seconds, and therefore, it is determined that the electronic objects 932 and 933 are the synchronous display type.

  According to the above processing, the conference terminal of the present invention can determine the object type according to whether or not the physical object that is the basis of each electronic object is stationary in the captured image. As a result, for a physical object that can be fixed on the imaging surface with a magnet or the like, a synchronous display type object type can be assigned to the corresponding electronic object. A physical object such as a hand can be given a permanent display type object type by a gesture that is excluded from the imaging surface before reaching the synchronization determination threshold.

<Update execution judgment>
The update execution determination unit 352 determines whether to perform two update processes for each electronic object. One is a process of updating the electronic object information in the electronic object storage unit 331 on the terminal side performed by the local electronic object storage unit 313. The other is processing for updating the electronic object information in the electronic object storage unit 331 of the opposite terminal, which is performed via the electronic object packet transmission unit 314.

  FIG. 10 is a flowchart of a process for performing update execution determination for each electronic object performed by the update execution determination unit 352. The update execution determination unit 352 determines whether the registration is new based on whether or not equivalent information is stored in the electronic object storage unit 331 (S1004). If it is a new registration, the update execution determination unit 352 branches the process according to the object type of the electronic object (S1005). If the object type is the synchronous display type or the permanent display type, the update execution determining unit 352 sets the transmission flag to TRUE (S1012) and sets the storage flag to TRUE (S1014). If the object type is an undetermined type, the update execution determining unit 352 sets the transmission flag to FALSE (S1022) and sets the storage flag to TRUE (S1024). That is, an electronic object that is still newly registered is unconditionally registered. However, if the object type is undetermined, it is not transmitted to the opposite terminal. This is because when the displayed time is short, such as 4 seconds or less, it is often unnecessary to reflect it on the opposite terminal.

  If it is not a new registration, it is determined whether the image information and the matching information of the electronic object have changed since the previous registration (S1008). If it has changed, the process branches according to the object type (S1009). If the object type of the electronic object is the synchronous display type, the update execution determining unit 352 sets the transmission flag to TRUE (S1032) and the storage flag to TRUE (S1034). If it is a permanent display type, the update execution determination unit 352 sets the transmission flag to FALSE (S1042) and the storage flag to FALSE (S1044). If the type is undetermined, the transmission flag is changed to FALSE (S1052), and the storage flag is changed to TRUE (S1054). In other words, an electronic object that has changed from an already registered one transmits and stores a change if it is a synchronous display type, but neither transmits nor stores a change if it is a permanent display type. The content of the undetermined type electronic object is updated in the terminal, but is not transmitted to the opposite terminal.

  If not moved, the update execution determination unit 352 determines whether there is a change in the object type or physical presence / absence from the electronic object stored in the electronic object storage unit 331 (S1010). If there is no change, the update execution determination unit 352 sets the transmission flag to FALSE (S1062) and sets the storage flag to FALSE (S1064). If there is a change in the object type, the update execution determination unit 352 sets the transmission flag to TRUE (S1072) and the storage flag to TRUE (S1074). That is, when the information has not been updated since the previous shooting, the data amount can be suppressed because the transmission process and the update process are not performed.

  If the transmission flag is TRUE (S1082), the update execution determination unit 352 adds the electronic object to the transmission candidate (S1084). If the storage flag is TRUE (S1092), the update execution determination unit 352 adds the electronic object to the storage candidate (S1094). If there is another electronic object to be processed, the process is repeated (S1099). If there is no electronic object to be processed, the transmission candidate is transmitted to the opposite terminal by notifying the electronic object packet transmission unit 314 of the transmission candidate (S1086). In addition, the electronic object information stored in the electronic object storage unit 331 is updated by notifying the local electronic object storage unit 313 of the storage candidate (S1096).

  The update execution determination means will be specifically described based on FIG. First, consider the stage of FIG. Although the electronic object 901 is newly registered, since the object type is an undetermined type, the transmission flag is set to FALSE, and the storage flag is set to TRUE (S1022, S1034). That is, the information is stored in the terminal but is not transmitted to the opposite terminal.

  Consider the stage of FIG. As for the electronic object 911, the electronic object 901 existing in the same position with the same image information is already registered, and the position is not moved. Since the object type has changed from the undetermined type to the permanent display type, the transmission flag is set to TRUE, and the storage flag is set to TRUE (S1072, S1074). Therefore, the information is stored in the terminal and information is transmitted to the opposite terminal. On the other hand, since the electronic objects 912, 913, and 914 are newly registered and the object type is an undetermined type, the transmission flag is FALSE and the storage flag is TRUE.

  Consider the stage of FIG. As for the electronic object 921, the electronic object 901 has already been registered, and the position has not moved. However, since the object type has changed from the permanent display type to the synchronous display type, the transmission flag is set to TRUE and the storage flag is set to TRUE. Therefore, the information is stored in the terminal and information is transmitted to the opposite terminal. In the electronic objects 922 and 924, the electronic objects 912 and 914 are registered, and the position is not changed but the object type is changed. Therefore, the transmission flag is set to TRUE and the storage flag is set to TRUE. Since the electronic object 923 is not newly added, the position information is not changed, and the physical presence / absence information is none, the transmission flag is set to TRUE and the storage flag is set to TRUE. In other words, what is excluded by this is shared.

  Finally, as for the electronic object 931, the electronic object 901 has already been registered, and there is no change in both position and object type. Therefore, both the transmission flag and the storage flag are set to FALSE. Since the electronic objects 932 and 933 are registered with the electronic objects 922 and 924 and the position is not changed but the object type is changed, the transmission flag is set to TRUE and the storage flag is set to TRUE.

<Formation of projected image>
The projection image forming unit 332 forms a projection image to be projected by the projection image output unit 333 using the image projection module 212 based on the electronic object stored in the electronic object storage unit 313.

  FIG. 11 is a flowchart for forming a projection image from an electronic object. FIG. 12 is an example of an electronic object stored in the electronic object storage unit 331. FIG. 13 is an example of forming a projection image from an electronic object.

  The projection image forming unit 332 initializes the projection image with blank paper (S1102). Then, the projection image forming unit 332 repeats the following process for all the electronic objects stored in the electronic object storage unit 313. Processing branches depending on the object type of the electronic object (S1104). If the object type of the electronic object is a deleted type, the projection image forming unit 332 sets the image drawing flag to FALSE (S1110).

  If the object type of the electronic object is an undetermined type, the projection image forming unit 332 sets the image drawing flag to FALSE (S1112). That is, if the object type is neither the permanent display type nor the synchronous display type, the electronic object is not displayed. When the object type of the electronic object is a permanent display type, the projection image forming unit 332 sets the image drawing flag to TRUE (S1122). That is, when the object type is a permanent display type, the electronic object is displayed on both the present terminal and the opposite terminal.

  When the object type of the electronic object is the synchronous display type, the projection image forming unit 332 determines whether the electronic object generation terminal information matches that of the terminal (S1106). If they match, the projection image forming unit 332 sets the image drawing flag to FALSE (S1132). This is because if the electronic object is generated by the terminal, the physical object is displayed on the screen, and thus it is not necessary to project it. If not, it is determined whether a physical object exists (S1108). When the physical object exists, the projection image forming unit 332 sets the image drawing flag to TRUE (S1142). If the physical object does not exist, the projection image forming unit 332 sets the image drawing flag to FALSE (S1152). That is, when the object type is the synchronous display type, the electronic object generated by this terminal is not drawn. In addition, an electronic object that is generated by the opposite terminal but has no physical object is not displayed. An electronic object that is generated by the opposite terminal and has a physical object is displayed.

  As a result, the projection image forming unit 332 determines the image drawing flag (S1162), and when TRUE is set, the image is drawn on the projection image based on the position information and image information of the electronic object (S1164). If there is another electronic object to be processed, the process is repeated, otherwise the process ends (S1199).

  A specific example of projection image formation will be described with reference to FIGS. FIG. 12A shows the arrangement of physical objects on the imaging plane at the base where the terminal 101 is installed. FIG. 12B shows the arrangement of physical objects on the imaging plane at the base where the terminal 102 is installed. FIG. 12C shows an electronic object stored in the electronic object storage unit 331 on the terminal 101 side. FIG. 12D shows an electronic object stored in the electronic object storage unit 331 on the terminal 101 side.

  Electronic objects 1251, 1252, 1253, 1254, and 1258 are electronic objects generated on the terminal 101 side. In the electronic object 1253, the original physical object is already excluded and does not exist on the photographing surface, and the object type is a permanent display type. In the electronic object 1258, the physical object 1208 that is the source of the electronic object 1258 exists on the photographing surface, but the object type is set to the undetermined type. Electronic objects 1255, 1256, and 1257 were generated on the terminal 102 side. In the electronic object 1257, the original physical object is already excluded and does not exist on the photographing surface, and the object type is a permanent display type.

  Different projected images are formed on the terminals 101 and 102, respectively, and are superimposed on the photographing surface. A specific example is shown in FIG. FIG. 13A shows an example of a projected image formed by the terminal 101 based on the electronic object group shown in FIG. Since the electronic objects 1253 and 1257 are of a permanent display type, an image is drawn on the projection image to be projected by the terminal 101. The electronic objects 1255 and 1256 are synchronous display types, and are generated by the opposite terminal. Since a physical object exists on the photographing surface of the opposite terminal, an image is drawn on the projection image to be projected by the terminal 101. The Since the electronic object 1258 has an undefined object type, the electronic object 1258 is not displayed on the projection image to be projected by the terminal 101. The projection image FIG. 13A is formed by the above processing. When this projection image is superimposed and displayed on the imaging surface of FIG. 12A, FIG. 13C is obtained.

  FIG. 13B is an example of a projection image formed by the terminal 102 based on the electronic object group illustrated in FIG. Since the electronic objects 1253 and 1257 are of a permanent display type, an image is drawn on the projection image to be projected by the terminal 102. The electronic objects 1251, 1252, and 1254 are synchronous display types, and are generated by the opposite terminal. Since a physical object exists on the photographing surface of the opposite terminal, an image is drawn on the projection image to be projected by the terminal 102. . The projection image FIG. 13B is formed by the above processing. When this projection image is superimposed and displayed on the imaging surface of FIG. 12B, FIG. 13D is obtained.

  In this way, the same information can be shared between the terminals by forming a projection image based on the electronic object at each terminal and displaying the projection image superimposed on the physical object on the photographing surface. At this time, the permanent display type electronic object is displayed on any terminal even if no physical object exists. The synchronous display type electronic object is displayed when a physical object exists. Information on an electronic object whose object type is undetermined is not shared between the conference terminals.

  As described above, according to the information processing apparatus of the present invention, when an electronic object is shared with an information processing apparatus other than its own apparatus, the operability can be improved by specifying the type of the electronic object and switching the sharing method. It becomes. The electronic object of the synchronous display type reflects the change in the electronic object on the projection surface with respect to the change in the physical object on the imaging surface. Accordingly, an operation such as movement or exclusion of an electronic object via a physical object, that is, a tangible interface can be introduced into a conference place, and operability for conference participants can be improved. In addition, the electronic object of the permanent display type does not reflect the change in the electronic object on the projection surface with respect to the change in the physical object on the imaging surface. In this way, various materials that can be photographed by the camera can be taken into the conference without worrying about physical restrictions. As a result, it is possible to solve the complexity of preprocessing when taking materials into the meeting place, and to greatly reduce the load of preparation at the time of holding the meeting.

(Example 2)
In the second embodiment, description will be made centering on object type assignment manually performed by a conference participant. Even with the means of the first embodiment, it is not always possible to reliably estimate the object type of the electronic object desired by the conference participant from the still image / moving image characteristics of the partial area. As long as it is an estimate, it is reasonable to consider that certain misrecognitions are included. Therefore, there is a need for a way to recover misrecognition of object types in some way. For example, a specific object type must be set. Alternatively, it is necessary that the conference participant can easily change the object type of the electronic object later.

  As one recovery means for erroneous recognition of the object type, a means for explicitly specifying the object type of the electronic object generated from the captured image acquired in the next shooting can be considered. FIG. 14 shows an example of a user interface that explicitly designates an object type and a display on the projection plane at that time. The setting dialog 1400 is a dialog box for forcibly giving an explicit object type displayed on the projection plane. This is displayed by drawing by the projection image forming unit 332 according to instructions from the gesture input unit 361, the HID input unit 362, and the voice input unit 363.

  A setting dialog 1400 is a selection item for setting an object type determination mode. In this example, as the object type determination mode, an automatic determination mode, a fixed determination (permanent) mode, and a fixed determination (synchronous) mode can be selected. When the user designates the automatic determination button 1402, the object type determination mode becomes the automatic determination mode. When the user designates a fixed determination (permanent) button 1406, the object type determination mode becomes a fixed determination (permanent) mode. When the user designates a fixed determination (synchronization) button 1408, the object type determination mode becomes a fixed determination (synchronization) mode.

  When the object type determination mode is the automatic determination mode, the object type of the electronic object described above is determined based on at least one of still image characteristics and moving image characteristics of the partial area from which the object type is determined. When the object type determination mode is the fixed determination (synchronous) mode or the fixed determination (synchronous) mode, the object determination when the next electronic object is captured is fixedly determined.

  Note that it may be possible to specify which mode is to be set by the gesture input unit 361 and the voice input unit 363 without using the setting dialog 1400. For example, the object type determination mode is set to a fixed determination (permanent) mode when a gesture of rotating the right arm clockwise is performed, and is set to a fixed determination (synchronization) mode when a gesture of rotating counterclockwise is performed. It is also possible.

  Furthermore, it may be possible to designate which mode is to be selected at the same time that the photographing timing is instructed via the gesture input unit 361 and the voice input unit 363. When the voice input unit 363 recognizes that the voice input from the voice input module 213 is “shooting with a permanent display type”, the electronic object is captured after the object type determination mode is set to the fixed determination (permanent) mode. Execute the process. When the voice input unit 363 recognizes that the voice input from the voice input module 213 is “shooting with a synchronous display type”, the electronic object is captured after the object type determination mode is set to the fixed determination (synchronization) mode. Execute the process.

  At this time, in order to notify the conference participant of the object type determination mode, a means for displaying the object type determination mode, such as label buttons 1452, 1454, and 1456, may be separately provided on the projection plane.

  As one recovery means for erroneous recognition of the object type, a means for specifying the object type in a fixed manner according to the position where the physical object is placed can be considered. FIG. 15 is a flowchart for specifying an object type according to a position. FIG. 16 is a display example of a partial area in which the object type is designated according to the position.

  In the means for fixedly specifying the object type in accordance with the position where the physical object is placed, a forced permanent display area is provided in a partial area on the imaging surface. For example, as shown in FIG. 16A, the peripheral portion of the imaging surface can be a forced permanent display area 1601. The lower part can be a forced permanent display area 1611 as shown in FIG. As shown in FIG. 16C, the lower corners of the imaging surface can be used as forced permanent display areas 1621 and 1622. Alternatively, as shown in FIG. 16D, the left and right ends of the imaging surface can be set as forced permanent display areas 1631 and 1632.

  The forced permanent display area can use various shapes, and these are just one variation thereof. Further, the shape does not need to be rectangular, and various shapes such as a triangle, a circle, and an arc can be used. In addition to the condition that the electronic object partially overlaps the forced permanent display area, a condition that the electronic object is surely included in the forced permanent display area may be used. A forced permanent display area may be drawn on the projection image and superimposed on the projection plane.

  First, based on the contents of the electronic object storage unit 331, it is determined whether the electronic object generated by the electronic object generation unit 312 is stationary (S1502). If the electronic object type determining unit 351 is not stationary, the electronic object type determining unit 351 determines that the object type is an undetermined type (S1512). If the object is stationary, the stationary time is obtained (S1504). In step S1506, the electronic object type determination unit 351 determines that the object type is an undetermined type when the stationary time is equal to or less than the capture determination threshold that is an arbitrary threshold. If the stationary time is larger than the capture determination threshold, it is determined whether the electronic object is on the forced permanent display area (S1508). The electronic object type determining means 351 determines that the object type is an undetermined type if it is not on the area. If it is on the area, it is determined as the permanent display type (S1510). If there is another electronic object to be processed, the process is repeated; otherwise, the process ends (S1599). In this flowchart, if it is determined in step S1508 that it is not on the forced permanent display area, it is determined to be an undetermined type in step S1512. However, the display may be determined as the synchronous display type if it is displayed outside the partial area for a predetermined time, that is, when the stationary time is larger than the capture determination threshold.

  For example, when the physical object 1601 in FIG. 16A is captured, since the electronic object based on the physical object 1601 is on the forced permanent display area 1611, the object type of the electronic object is determined to be the permanent display type. The Note that a forced synchronous display area may be provided on the imaging surface in order to provide a synchronous display type.

  As one recovery means for erroneous recognition of the object type, a method of designating a change of the object type by placing a physical object on the electronic object is also conceivable. FIG. 17 is a flowchart for changing an object type by placing a physical object on an electronic object. FIG. 18 is an example of a flowchart for changing an object type by placing a physical object on an electronic object.

  First, based on the contents of the electronic object storage unit 331, it is determined whether the electronic object generated by the electronic object generation unit 312 is stationary (S1702). If the electronic object type determination unit 351 is not stationary, the electronic object type determination unit 351 determines that the object type is an undetermined type (S1712). If the object is stationary, the electronic object type determining unit 351 obtains the stationary time (S1704). In step S1706, the electronic object type determination unit 351 determines that the object type is an undetermined type when the stationary time is equal to or less than the synchronization determination threshold value which is an arbitrary threshold value. If the stationary time is larger than the synchronization determination threshold, it is determined whether the electronic object is on an electronic object having the same image information and position information (S1708). The electronic object type determination means 351 determines that the object type is an undetermined type if it is not above this. If it is above, it is set as a synchronous display type (S1710). If there is another electronic object to be processed, the process is repeated, otherwise the process ends (S1799).

  A specific operation will be described with reference to FIG. FIG. 18A shows a state in which the conference participant is bringing the physical object 1801 close to the electronic object 1811 having the permanent display type object type stored in the electronic object storage unit 313.

  FIG. 18B shows a state where the physical object 1801 is moved and stationary so that an electronic object having the same image information and position information is generated on the electronic object 1811. When an electronic object is generated from the physical object 1801, the electronic object 1811 stored in the electronic object storage unit 313 is updated from the permanent display type to the synchronous display type by the generated electronic object.

  FIG. 18C shows an example in which the physical object 1801 is moved. At this time, the electronic object is not displayed on the projection plane because it is a synchronous display type and has physics. For example, in this state, the conference participant uses the above-described recovery means to change the object type from the synchronous display type to the permanent display type and instruct to take in the electronic object via the voice input means 363. . Then, the object type of the electronic object 1811 is updated from the synchronous display type to the permanent display type.

  FIG. 18D shows an example in which the physical object 1801 is moved. The physical object 1801 and the electronic object 1811 are displayed independently as the object type is changed. As shown above, by using this function, it is possible to reset an electronic object imported as a permanent display type to an object type of synchronous display type with a simple operation of overlapping physical objects. Become.

(Example 3)
In the third embodiment, utilization of distance information for object type determination will be described. The conference terminal according to the third embodiment is characterized in that the object type of the electronic object is determined based on the still image / moving image characteristics of the partial area included in the captured image. However, for example, when a photograph taken with a hand or magnet is shown on a whiteboard, it is difficult to reliably distinguish whether the hand or magnet is actually a three-dimensional object or a plane that does not actually exist. For this reason, an erroneous determination of the object type may occur.

  In the electronic conference terminal of this embodiment, the image capturing module 211 is a stereo camera. A stereo camera is a camera that has two normal video acquisition cameras and estimates distance information with respect to a subject to be photographed from feature points of images taken by the respective cameras. It is determined from the distance information between the physical object on the imaging surface and the camera whether or not the imaging target is a three-dimensional object.

  Here, in order to acquire the distance information, a distance image sensor technique may be used instead of the stereo camera. The distance image sensor is a sensor that measures distance information with respect to an object to be imaged, and for example, an infrared reflection time is used. The distance image sensor and the stereo camera may be used for other purposes such as gesture recognition based on a three-dimensional model and super-resolution processing from a plurality of images.

  FIG. 19 shows an example of object type determination processing to which distance information is applied. On the imaging surface 1900, a physical object 1901 and a physical object 1911 are included. The physical object 1901 is fixed by magnets 1903 and 1902. On the other hand, the physical object 1911 is fixed by the hand 1912. A distance map 1950 is a result obtained by photographing the same area as the photographing surface 1900 with a distance image sensor or a stereo camera and obtaining distance information.

  In the distance map 1950, the shape 1961 can be observed on the distance information corresponding to the physical object 1911. On the other hand, the shape corresponding to the physical object 1901 cannot be observed. Based on this, it is possible to identify a physical object 1911 having a certain distance from the imaging surface and an object 1901 attached to the imaging surface. For example, a permanent display type can be assigned to an electronic object generated from a partial area obtained by photographing the physical object 1911. Further, a synchronous display type can be assigned to a physical object generated from a partial area where the physical object 1901 is photographed. That is, an electronic object that is determined to be in a position before the shooting surface is determined to be a permanent display type, and an electronic object that is determined to be substantially the same as the shooting surface is determined to be a synchronous display type. The

  In the distance map 1950, the shapes 1952 and 1953 can be observed on the distance information so as to correspond to the magnets 1902 and 1903. By recognizing that the magnet is a “three-dimensional object” instead of “printed material”, the object type is determined as the synchronous display type only when the partial area includes a magnet image that is a three-dimensional object. I can do things.

  Further, the shape 1962 can be observed in the distance map 1950 so as to correspond to the hand 1912. A hand or arm can be expected to have a slight roundness, and from this, it can be expected that gradation is also formed in distance information. Based on this, the hand portion can be detected with higher accuracy than the discrimination using only the color. As described above, by using the distance information for determining the object type, the recognition accuracy of the object type can be further increased.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (computer program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media. Then, the computer (or CPU, MPU, etc.) of the system or apparatus reads out and executes the program. In this case, the program and the storage medium storing the program constitute the present invention.

  101, 102 Conference terminal

Claims (18)

An information processing apparatus that shares captured image information among a plurality of information processing apparatuses via a network,
Generating means for extracting a physical object from image information captured using the projection plane as an imaging plane, and generating an electronic object corresponding to the physical object;
Receiving means for receiving an electronic object from an information processing apparatus other than the own apparatus;
Projection means for projecting a projection image generated based on the generated or received electronic object onto the projection plane;
Based on the characteristics of the image area corresponding to the physical object, the type of the generated electronic object is projected by the information processing apparatus other than the own apparatus in synchronization with the change of the physical object on the photographing surface of the own apparatus. A determination means for determining whether the synchronization type is drawn on the image or the permanent type to be permanently projected onto the projection plane of the information processing apparatus other than the own apparatus without being synchronized;
Transmission means for transmitting the electronic object determined to be the synchronization type or the permanent type to an information processing apparatus other than the own apparatus,
When the change occurs in the physical object, the transmission means transmits information about the electronic object to an information processing apparatus other than the own apparatus when the electronic object is the synchronization type, and the electronic object When the permanent type, the information processing apparatus does not transmit information regarding the electronic object. When the change occurs in the physical object, the projection unit projects the generated projection image on the projection plane without performing an update process of the electronic object when the electronic object is the permanent type, The information processing apparatus according to claim 1, wherein when the electronic object is the synchronous type, a projection image generated by performing update processing is projected onto the projection plane. Further comprising drawing means for drawing the projection image based on the generated or received electronic object;
When the drawing means is the synchronous type and is an electronic object generated by the generation means of its own apparatus, the electronic object received from an information processing apparatus other than the own apparatus is not drawn as the projection image. The information processing apparatus according to claim 1, wherein when the object is an object, the electronic object is drawn as the projection image. The determining means determines the type of the electronic object based on a still image or a moving image,
When the still image is used, when the first object that holds the physical object is included in the still image, it is determined as the permanent type, and the second object that holds the physical object in the still image is If it is included, the sync type is determined,
When the moving image is used, it is determined that the synchronization type is the stationary type when the stationary time of the electronic object exceeds a first predetermined time, and is the permanent type when the electronic object does not exceed the first predetermined time. The information processing apparatus according to claim 1, wherein the information processing apparatus is determined. The information processing apparatus according to claim 4, wherein the first object is a hand, and the second object is at least an identification marker, a magnet, or a tag. The determination means determines that the electronic object is of a permanent type when the electronic object does not exceed the first predetermined time but exceeds the second predetermined time, and the electronic object is the second type. 6. The information processing apparatus according to claim 5, wherein when the predetermined time is not exceeded, the electronic object is determined to be an undetermined type. When the electronic object is newly generated and is of the synchronization type or the permanent type, the transmission means transmits information about the electronic object to an information processing apparatus other than its own device, and the electronic object The information processing apparatus according to claim 6, wherein when it is the undetermined type, information about the electronic object is not transmitted to an information processing apparatus other than its own apparatus. The transmission means, when the type of the electronic object is changed from the permanent type to the synchronization type, transmits information about the electronic object to an information processing apparatus other than the own apparatus. 8. The information processing apparatus according to 7. When a physical object having the same content as the electronic object is photographed in a state where the electronic object of the permanent type is projected on the projection plane, both objects are displayed on the projection plane of the own apparatus. The information processing apparatus according to any one of claims 1 to 8. When the physical object is displayed on the photographing surface, the display device further includes a display unit that displays on the projection surface a selection item for selecting whether the electronic object is the synchronous type or the permanent type. The information processing apparatus according to any one of claims 1 to 3. The projection plane is provided with one or a plurality of partial areas for determining the permanent type when the electronic object is displayed,
The determination means determines that the electronic object is a permanent type when the electronic object is displayed in the partial area beyond the second predetermined time, and exceeds the second predetermined time other than the partial area. The information processing apparatus according to any one of claims 1 to 3, wherein the electronic object is determined to be an undetermined type. When the physical object having the same content as the electronic object is displayed on the electronic object that is determined to be the permanent type projected on the projection plane, the determination unit is configured to synchronize the electronic object with the synchronization object. The information processing apparatus according to claim 1, wherein the information processing apparatus is changed to a type. The projection plane is provided with one or a plurality of partial areas for determining the permanent type when the electronic object is displayed,
The determination means determines that the electronic object is a permanent type when the electronic object is displayed in the partial area beyond the second predetermined time, and exceeds the second predetermined time other than the partial area. The electronic object is determined to be a synchronous type when it is displayed, and is determined to be an undetermined type when displayed without exceeding the second predetermined time. The information processing apparatus according to claim 1. A distance image sensor for measuring distance information with respect to the subject;
The determination unit determines whether the electronic object is the synchronous type or the permanent type based on distance information measured by the distance image sensor. The information processing apparatus according to item. The information processing apparatus according to claim 1, further comprising: a voice input unit that receives an instruction for determining a type of the electronic object, a gesture input unit, or a human interface device. . When the second object is the sticky note, the determination unit extracts the color of the sticky note and the background color of the physical object, and if the color of the sticky note matches the background color, it is the synchronization type. The information processing apparatus according to claim 5, wherein the information processing apparatus determines that the type is a permanent type if they do not match. An information processing apparatus control method for sharing captured image information among a plurality of information processing apparatuses via a network,
A generation step of extracting a physical object from image information captured using the projection plane as an imaging plane, and generating an electronic object corresponding to the physical object;
A receiving step of receiving an electronic object from an information processing apparatus other than the own apparatus;
A projection step of projecting a projection image generated based on the generated or received electronic object onto the projection plane;
Based on the characteristics of the image area corresponding to the physical object, the type of the generated electronic object is projected by the information processing apparatus other than the own apparatus in synchronization with the change of the physical object on the photographing surface of the own apparatus. A determination step of determining whether it is a synchronous type drawn on an image or a permanent type that is permanently projected onto a projection surface of an information processing device other than its own device without being synchronized;
A transmission step of transmitting the electronic object determined to be the synchronization type or the permanent type to an information processing apparatus other than the own apparatus,
In the transmission step, when a change occurs in the physical object, when the electronic object is the synchronization type, information on the electronic object is transmitted to an information processing apparatus other than the own apparatus, and the electronic object When the permanent type, the control method is characterized in that information on the electronic object is not transmitted. A computer program for causing a computer to execute the control method according to claim 17.

Images