JP5502771B2 - Electronic blackboard system and program - Google Patents

Description

  The present invention relates to an electronic blackboard system capable of detecting, as a two-dimensional coordinate on a display surface, the contact position of the coordinate indicator on the display surface or the operation position of the coordinate indicator on the detection surface set on the near side of the display surface. The present invention relates to a program that is executed on a computer constituting the computer.

  The electronic blackboard system includes a display device or a projection device, a coordinate detection device that detects an operation input of a coordinate indicator on the display surface, a computer that controls a drawing process based on the detected operation input and an operation of the entire system, It consists of

  The coordinate detection device outputs the detection position as two-dimensional coordinates on the display screen. The computer converts the two-dimensional coordinates of the coordinate detection device into two-dimensional coordinates on the computer operation screen, and draws the object on the computer operation screen as an object having a color and thickness designated in advance. This computer operation screen is provided to the user as a display surface of the display device or the projection device.

  The electronic blackboard system repeats the process from the coordinate detection by the coordinate detection device to the drawing on the computer operation screen dozens of times per second, so that the operator of the coordinate pointing object looks like a character or figure on the computer operation screen. It is possible to provide the user with the experience of writing directly. In addition, when the electronic blackboard system supports the function of handling the operation input by the coordinate indicator equivalent to the operation of the mouse connected to the computer, the computer may be operated through the operation input of the coordinate indicator on the display surface. it can.

JP 2001-67183 A

  By the way, the coordinate detection method of the coordinate detection device used in the electronic blackboard system includes a position (operation position) where a predetermined or arbitrary coordinate indicator has entered a virtual detection surface set within a range of several centimeters from the display surface. ) Is detected optically.

  For example, Patent Document 1 discloses a coordinate detection device including a camera configured by a color two-dimensional imaging device and a lens having an angle of view capable of photographing the entire detection surface at both upper ends of a display surface. This coordinate detection apparatus (1) calculates the difference between a reference image captured by two left and right cameras in advance when the coordinate indicator is not on the detection surface and the latest image captured by the two cameras thereafter for both left and right images. (2) When the coordinate pointing object is on the detection surface, the respective directions from the left and right cameras are obtained from the position of the silhouette remaining in the difference image, and the distance between the cameras that has been measured in advance is used. Then, the process of calculating the coordinates of the coordinate indicator on the display surface in the manner of triangulation is executed. In addition, in order to reduce the generation of noise due to ambient light, etc., a background plate for narrowing the field of view is arranged at the peripheral portion except the upper part of the display surface, and a vertically long shape is used to limit the imaging region to the detection surface. It is disclosed that a light shielding plate having a notch (slit) is attached in front of the imaging surfaces of the left and right cameras. With these background plate and light shielding plate, the two cameras on the left and right images only the background plate if there is no coordinate indicator.

  However, when the size of a practical electronic blackboard is reached (for example, when the aspect ratio of the display surface is 4: 3 or 16: 9 and the diagonal length is about several tens of inches or more), the coordinate detection device of Patent Document 1 is used. Then, it becomes difficult to continue to limit the shooting range of the camera to only the background plate due to the influence of the distortion of the casing accompanying the temperature change and the secular change. Specifically, the scenery outside the display surface and the background plate is reflected in the imaging range. In order to correct this inconvenience, it is necessary to finely adjust the mounting position of the camera and the light shielding plate by the user and support staff.

  Therefore, the present invention provides an electronic blackboard system that can always acquire only the image of the background plate and can accurately calculate the coordinates of the coordinate indicator without performing fine adjustment work of the mounting position of the camera and the light shielding plate. The purpose is to provide a program that realizes.

The inventor proposes a mechanism for automatically identifying only the area of the background plate from the image captured without using the light-shielding plate and detecting the coordinates of the coordinate indicator using only the image of the identified area. To do. More specifically, the present inventor proposes an electronic blackboard system having the following means.
(a) Display surface designation screen display control means for displaying a display surface designation screen including one or more display surface designation targets for designating the contact position of the coordinate pointing object on the display surface having a predetermined background plate on the outer periphery
(b) one or more image pickup means for picking up an image of a coordinate indicator inserted on a detection surface near the sky of the display surface;
(c) a reference image captured by the imaging unit when no coordinate indicator is inserted on the detection surface, and a target image captured by the imaging unit when a coordinate indicator is inserted on the detection surface To target coordinate detection means for detecting target coordinates corresponding to the display surface indication target in the captured image of the imaging means
(d) First edge image generation means for generating a first edge image obtained by extracting an edge of the reference image
(e) In the first edge image, adjacent parallel edges are synthesized to generate an edge synthesis area, and edges that do not satisfy the first relative positional relationship with respect to the target coordinates are removed to remove edge synthesis. Edge synthesis removed image generating means for generating an image
(f) Second edge image generation means for generating a second edge image obtained by extracting an edge of the edge synthesis removed image
(g) Background board area extracting means for extracting an edge satisfying a second relative positional relationship with respect to the target coordinates as a background board area corresponding to the outline of the background board in the second edge image.

The inventor also proposes a program that causes a computer constituting the electronic blackboard system to function as the following means.
(a) Display surface designation screen display control means for displaying a display surface designation screen including one or more display surface indication targets for instructing the contact position of the coordinate indicator on the display surface having a predetermined background plate on the outer periphery.
(b) Control means for controlling one or more image pickup means for picking up an image of a coordinate indicator inserted on a detection surface near the sky on the display surface
(c) a reference image captured by the imaging unit when no coordinate indicator is inserted on the detection surface, and a target image captured by the imaging unit when a coordinate indicator is inserted on the detection surface To target coordinate detection means for detecting target coordinates corresponding to the display surface indication target in the captured image of the imaging means
(d) First edge image generation means for generating a first edge image obtained by extracting an edge of the reference image
(e) In the first edge image, adjacent parallel edges are synthesized to generate an edge synthesis area, and edges that do not satisfy the first relative positional relationship with respect to the target coordinates are removed to remove edge synthesis. Edge synthesis removed image generating means for generating an image
(f) Second edge image generation means for generating a second edge image obtained by extracting an edge of the edge synthesis removed image
(g) Background board area extracting means for extracting an edge satisfying a second relative positional relationship with respect to the target coordinates as a background board area corresponding to the outline of the background board in the second edge image.

  By adopting the present invention, an electronic blackboard system that not only eliminates the need for a light-shielding plate that limits the imaging range, but also enables accurate calculation of the coordinates of the coordinate indicator without performing fine adjustment work such as the camera mounting position. can do. Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments.

The figure explaining the example of apparatus arrangement | positioning around an electronic blackboard. The figure explaining the example of a connection of a computer and peripheral devices. The figure which shows the process sequence of a background board area | region extraction function part. The figure which shows an example of a display surface designation | designated screen. The figure which shows an example of a whole image. The figure which shows an example of a 1st edge image. The figure which shows an example of an edge synthetic | combination removal image. The figure which shows an example of the method of approximating the boundary line of a background board area | region with a function in a 2nd edge image.

  Embodiments of the invention will be described below with reference to the drawings. In addition, all the electronic blackboard systems mentioned later are examples for description, and embodiment of invention is not limited to the Example mentioned later. In addition, the present invention is not limited to combining all the functions of the electronic blackboard system to be described later, but when only a part of the functions is mounted, when adding a known technique, the partial function is replaced with the known technique. Various embodiments can be included, such as In the following embodiments, various processing functions are described as provided as programs executed on a computer. However, some or all of the processing functions may be realized through hardware.

(Configuration of electronic blackboard)
FIG. 1 shows an embodiment of the electronic blackboard 100. Note that the electronic blackboard 100 illustrated in FIG. 1 represents a case where an optical coordinate detection method is employed. In this example, infrared light is used. The electronic blackboard 100 shown in FIG. 1 uses a projector for displaying images. That is, an electronic blackboard system that combines an optical coordinate detection function and a projection image display function will be described.

  The electronic blackboard 100 has a configuration in which a camera A103, a camera B104, and a background plate 105 are attached to a flat electronic blackboard casing (board 101). These accessories are all arranged on the outer peripheral portion of the board 101. The central portion of the board 101 is used as the display surface 102 onto which the projection light from the projector 210 (FIG. 2) is projected. Of course, when a flat panel display is used to display an image, the display surface 102 matches the display area of the display.

  Camera A103, camera B104, and background plate 105 constitute a hardware part of the coordinate detection apparatus. In this embodiment, the camera A 103 is arranged at the upper left corner of the board 101 and the camera B 104 is arranged at the upper right corner of the board 101. This is because a camera having an angle of view of about 90 ° is used. Accordingly, the camera A103 arranged in the upper left corner images the background plate 105 located on the lower side and the right side of the display surface 102, and the camera B104 arranged in the upper right corner is located on the lower side and the left side of the display surface 102. The background plate 105 to be captured is imaged.

  Note that when a camera with an angle of view larger than 90 ° (for example, about 180 °) can be used, the camera A 103 and the camera B 104 are located on the inner side of the left and right ends of the display surface 102 (shorter than the horizontal side length). Can be placed at a distance).

  The background plate 105 is installed perpendicular to the surface of the board 101 so as to follow the three sides of the left side, the right side, and the lower side of the board 101 having a rectangular shape. The background plate 105 has a sufficient width to be captured as the background of the coordinate indicator when the camera A 103 and the camera B 104 capture the coordinate indicator. In addition, it is preferable that the surface of the background board 105 imaged with the camera A103 and the camera B104 is a striped pattern or a lattice pattern composed of a solid achromatic color or only an achromatic color.

  Further, a tape-like member having, for example, a reflexive reflection characteristic is disposed on the surface. By arranging the retroreflective member along the surface of the background plate 105, the light (for example, infrared light) incident on the surface of the background plate 105 from the camera side can be changed regardless of the difference in the incident angle. Also reflects in a direction parallel to the incident angle. For this reason, the surface of the background plate 105 is captured as a bright image, and a coordinate indicator (a person's finger, hand, pen, stylus, etc.) crossing the surface of the background plate 105 is captured as a shadow.

  The camera A 103 and the camera B 104 detect a coordinate indicator that approaches or touches the vicinity of the display surface 102 and two-dimensional coordinates that specify the position of the detected coordinate indicator on the display surface. Used. The camera A 103 and the camera B 104 image the background plate 105 based on an instruction from the PC 200 (FIG. 2). As a feature of the camera A103 and the camera B104 according to this embodiment, a slit that prevents imaging is not disposed in front of the imaging surface. For this reason, as will be described later, not only the background plate 105 but also all objects located in the vicinity thereof are imaged on the imaging screens of the camera A 103 and the camera B 104.

  Note that the output of the captured image from the camera A 103 and the camera B 104 to the PC 200 (FIG. 2) is preferably a digital signal. This is because the signal processing of the PC 200 (FIG. 2) is performed on the digital signal. Of course, if the PC 200 (FIG. 2) has a function capable of converting an image signal in the analog signal format into an image signal in the digital signal format, there is no problem with the output using the analog signal. Further, the image sensor of the camera may be a CMOS, CCD, or other image sensor as long as the above conditions are satisfied. Furthermore, the camera A 103 and the camera B 104 have a lens having an angle of view that can capture the display surface 102. Note that the camera A 103 and the camera B 104 are not limited to a single camera, and may be a collection of cameras having a small angle of view.

  The detection of the coordinates of the coordinate indicator is generally performed as follows. First, the camera A 103 and the camera B 104 respectively capture a left reference image and a right reference image based on an instruction from the PC 200 (FIG. 2). Note that when acquiring the reference image, it is essential that there is no coordinate pointing object in the imaging region. The captured reference image is temporarily stored in the RAM 203 (FIG. 2) until a new reference image is captured.

  Subsequently, the camera A 103 and the camera B 104 respectively capture the latest left image and the latest right image based on a periodic instruction from the PC 200 (FIG. 2). Next, the PC 200 (FIG. 2) generates a left difference image from the difference between the values of the respective pixels of the left reference image and the latest left image. Similarly, the PC 200 generates a right difference image from the difference between each pixel of the right reference image and the latest right image.

  When the left and right latest images are captured, if there is a coordinate indicator near the sky above the display surface 102, the silhouette remains in the left difference image and the right difference image. The PC 200 detects the position of the silhouette and detects angle information between the detected position and the mounting position of each camera. Thereafter, the PC 200 calculates the coordinates of the coordinate indicator on the display surface 102 according to the principle of triangulation based on the relationship between the detected angle information and the distance between the cameras.

  From the viewpoint of the stability of the coordinate values, it is necessary for the difference image used for calculating the coordinates not to be affected by slight fluctuations in the light environment from the viewpoint of the stability of the coordinate values. Therefore, instead of using a simple difference between the reference image and the latest image as the difference image, it is desirable to extract only the difference component equal to or greater than a certain value (threshold value) as the difference image.

  In addition, it is desirable to update the left and right reference images as needed to adapt to changes in the surrounding light environment. For example, when no silhouette remains in the left and right difference images (when the PC 200 detects that no silhouette remains), the left and right latest images may be replaced with existing reference images. Alternatively, when no silhouette remains in each of the left and right difference images (when the PC 200 detects that no silhouette remains), an average value of each pixel is calculated for a plurality of latest latest images, and the average value is calculated. You may use the image comprised by these as a reference | standard image.

  In the case of the present embodiment, the reference image, the latest image, and the like described above do not mean the whole image captured by the camera A 103 and the camera B 104 but a background plate region (reference image) extracted by a background plate region extraction function described later. It is used in the sense of a partial image corresponding to a partial image region cut out from the above.

(Internal configuration of computer)
FIG. 2 shows the internal configuration of the PC 200 connected to the electronic blackboard 100 and the connection between peripheral devices. The PC 200 includes a CPU 201, ROM 202, RAM 203, HDD 204, video card 205, display interface (I / F) 206, external interface (I / F) 207, and internal bus 208.

  The CPU 201 is an arithmetic device that executes a BIOS program, an OS, and an electronic blackboard program (including not only a coordinate position detection function but also a background plate area extraction function described later). The ROM 202 is an area for recording a BIOS program which is an operation procedure at the time of starting the PC 200 and a basic control procedure of each internal and external hardware of the PC 200. The RAM 203 is an area for temporarily storing a BIOS program, an OS, an electronic blackboard program, other programs, images taken by the camera A 103 and the camera B 104, processing results thereof, and the like. The HDD 204 is an area for recording the OS and the electronic blackboard program. The video card 205 is a device that calculates display data, temporarily stores it, and outputs it to a display device (a projector 210 in FIG. 2). A display I / F 206 is an interface for connecting the video card 205 and a display device. The external I / F 207 is an interface for connecting various devices constituting the electronic blackboard 100 to the PC 200. The internal bus 208 is used for data exchange between 201-207.

  In the PC 200, the operation is executed as follows. When the power of the PC 200 is turned on, the CPU 201 reads and executes the BIOS program read from the ROM 202. Subsequently, the CPU 201 copies the OS and the electronic blackboard program from the HDD 204 to the RAM 203 and executes them sequentially. When the CPU 201 executes an output command to the display device included in the BIOS program, the OS, and the electronic blackboard program, the CPU 201 sends the drawing command and drawing data to the video card 205. The video card 205 calculates drawing data based on a drawing command, temporarily accumulates the calculation results as display data, and then transmits and displays the display data as display data on a display device connected to the display I / F 206. When the CPU 201 executes an operation instruction of the electronic blackboard 100 included in the BIOS program, the OS, and the electronic blackboard program, the CPU 201 sends a predetermined instruction to the electronic blackboard 100 via the external I / F 207, receives the processing result of the instruction, and receives the RAM 203. Memorize temporarily.

  A projector 210 as a display device is connected to the PC 200 via a display cable 209 and projects display data sent from the PC 200 via the cable onto the display surface 102 of the electronic blackboard 100.

  The camera A 103 and the camera B 104 constituting the electronic blackboard 100 are connected to the PC 200 via the connection cable 211 and perform processing such as imaging based on a predetermined instruction sent from the CPU 201 via the cable, and the processing result Send back.

(program)
The BIOS program describes an operation procedure at the time of starting the PC 200 and a basic control procedure of each hardware inside and outside the PC 200. The OS is an operating system of the PC 200, abstracts the internal and external hardware of the PC 200, and operates each hardware based on instructions from the application program. For example, the OS causes the projector 210 to project display data, causes the camera A 103 and the camera B 104 of the electronic blackboard 100 to capture an image, and receives the captured image.

  The electronic blackboard program is an application program that runs on the OS, and extracts a partial area in which the background board 105 appears from images captured by the camera A 103 and the camera B 104 (background board area extraction function), and is captured by a pair of left and right cameras. For example, a process (coordinate position detection function) for calculating the designated coordinates of the coordinate designating object from each difference image between the reference image and the latest image is executed.

(Background board area extraction function)
Here, a suitable background board region extraction function will be described which is executed before using the electronic blackboard 100 (before starting operation input by the coordinate indicator, that is, before starting coordinate detection of the coordinate indicator). This function corresponds to calibration for accurately detecting the coordinates of the coordinate indicator. The function is preferably executed every time the electronic blackboard 100 starts to be used (for example, when the power is turned on), but once every several times, before the first use of the day, once every few days, It may be executed before the first use in the morning and before the first use in the afternoon.

  As described above, in the case of the camera A103 and the camera B104 used in this embodiment, the captured field of view of the captured image is not limited by a light shielding plate (slit) or the like. For this reason, not only the background plate 105 but also the board 101 (including the display surface 102) and the floor are imaged in the same frame on each imaging screen. Therefore, a background plate area extraction function is executed so that an area portion where only the background plate 105 is basically captured can be automatically extracted from images captured by the cameras A103 and B104. Fine adjustment of the (slit) mounting position is unnecessary.

  FIG. 3 shows an outline of a processing procedure of a program that realizes the background plate area extraction function. This program is automatically started when the electronic blackboard program is started, automatically at a timing satisfying a predetermined condition, or at the request of the user of the electronic blackboard system.

  In the following description, a case where a background plate region (image region of the background plate 105) included in the captured image (overall image) of the camera A103 is described will be described. However, it is included in the captured image (overall image) of the camera A104. The same processing procedure can also be applied when extracting a background plate area (image area of the background plate 105) to be extracted.

  In step 301, the background plate area extraction function unit (CPU 201) causes the projector 210 to display a display surface designation screen on the display surface 102, issues an imaging instruction to the camera A103, and receives the captured image. In step 301, the background plate area extraction function unit (CPU 201) operates as a display surface designation screen display control unit and a display surface instruction target coordinate detection unit.

  First, the function as the display surface designation screen display control unit will be described. The display surface designation screen is basically an entirely white screen prepared to facilitate the specification of the background plate area. However, in the display surface 102, one or more display surface instruction targets are arranged for instructing a user or support staff to specify a specific position with a coordinate indicator.

  Note that displaying the entire white screen (the screen having the same luminous intensity or illuminance over the entire screen) depends on the usage environment of the electronic blackboard 100, and external light is reflected on a part of the display surface 102. This is because uneven brightness may occur on the imaging screen. The luminance unevenness may be erroneously detected as the edge of the background plate 105. Therefore, the luminous intensity or illuminance here is set to a value that can erase the pseudo edge caused by the reflection from the imaging screen even if at least any external light exists.

  Further, the display surface instruction target is used to facilitate identification of an edge corresponding to the background plate 105 and other edges in the edge information included in the imaging screen in image processing, that is, the display surface 102 and the background plate 105. Displayed to facilitate identification of boundaries. FIG. 4 shows a display example of the display surface designation screen. In the case of the display surface designation screen shown in FIG. 4, one (total of four) display surface instruction targets are arranged near the four corners. Note that only one, two, three, five, or more display surface instruction targets may be arranged at arbitrary positions on the display surface designation screen. Preferably, one display surface instruction target is arranged on each side where the background plate 105 is installed. More preferably, two display surface indication targets are arranged on each side where the background plate 105 is installed.

  Next, the function as a display surface instruction | indication target coordinate detection part is demonstrated. The background plate area extraction function unit (CPU 201) outputs an imaging instruction to the camera A103 immediately after the display surface designation screen is displayed. As a result, the background board area extraction function unit (CPU 201) acquires a captured image (hereinafter referred to as “whole image”) from the camera A103.

  Similarly, the background board area extraction function unit (CPU 201) notifies the operator of the electronic blackboard 100 so that the display surface instruction target displayed on the display surface designation screen is indicated by the coordinate instruction object. For example, an instruction sentence is displayed on the display surface designation screen so that the display surface instruction target to be displayed on the screen is pointed to by a coordinate indicator. In addition, for advanced users who are used to the operation, a function of omitting the display of the instruction text and immediately displaying the display surface instruction target may be prepared.

  Even while the display surface designation screen is displayed on the display surface 102, the background plate area extraction function unit (CPU 201) outputs an imaging instruction and acquires a captured image (hereinafter referred to as “target image”) from the camera A103. .

  When a plurality of display surface instruction targets are displayed, the background plate area extraction function unit (CPU 201) displays the display surface instruction targets one by one on the screen. For example, in the case of FIG. 4, the background plate area extraction function unit (CPU 201) displays the display surface instruction target 1, then displays the display surface instruction target 2, and then displays the display surface instruction target 3. Next, the display surface indication target 4 is displayed. The background plate area extraction function unit (CPU 201) outputs an imaging instruction each time a new display surface instruction target is displayed on the screen, and sequentially acquires target images corresponding to the display surface instruction targets.

  Thereafter, the background plate area extraction function unit (CPU 201) uses the entire image as a reference image, and acquires the coordinate position of the display surface instruction target on the entire image from the difference image between the reference image and each target image. The coordinate positions here are the display surfaces in the image captured by the camera A103 and the image generated from the image (whole image, target image, difference image, first edge image, edge synthesis removed image, second edge image). Used as the coordinate position of the pointing target. Of course, for the camera B104 as well, the coordinate position of the display surface instruction target on the entire image is acquired and used as the coordinate position of each display surface instruction target in the image captured by the camera B104 and the image generated from this. It is done.

  FIG. 5 shows an example of the entire image captured by the camera A103. A bent and elongated region extending in the vertical direction in the center of the image in FIG. 5 corresponds to the background plate 105 positioned on the right side and the lower side of the display surface 102. The region is bent because the image is taken from a position offset from the surface formed by the two background plates 105. Further, a portion with a large region width means a region close to the camera A103, and a portion with a small region width means a region far from the camera A103. That is, it is the influence of the perspective effect.

  In addition, in the entire image shown in FIG. 5, the electronic blackboard housing (board 101) and the display surface 102 are imaged on the left side of the area corresponding to the background board 105, and the floor 106 is on the right side of the area corresponding to the background board 105. Imaged.

  This is because the portion of the area corresponding to the background board 105 that is slightly narrowed above the image corresponds to the lower right corner of the electronic blackboard 100. This is because the lower right corner of the electronic blackboard 100 hits the farthest place from the camera A103.

  FIG. 5 shows an example in which only the floor 106 is shown on the right side of the area corresponding to the background plate 105 for the sake of simplicity. However, in addition to the floor 106, people, objects, walls, windows, and the like may be reflected in the actual imaging screen. In the example of FIG. 5, the boundaries of the display surface 102, the background plate 105, the board 101, and the floor 106 are drawn as straight lines. However, in reality, due to the influence of the distortion of the lens attached to the camera A 103, etc. It is imaged slightly bowed (ie as an arc).

  For convenience of the following description, an image captured by the camera A103 and an image generated from the image (overall image, target image, difference image, first edge image, edge synthesis removed image, and second edge image) are all shown in the figure. The upper left corner is the coordinate origin, the right direction is the x-axis direction, and the lower direction is the y-axis direction.

  In step 302, the background plate area extraction function unit (CPU 201) creates a first edge image from the entire image. In the present specification, this function is referred to as a first edge image generation unit. The first edge image here corresponds to an image obtained by extracting only image data having a large change amount from the entire image. That is, for each pixel of the entire image, the first edge image is a value obtained by calculating an amount of change in brightness (primary differential value) from adjacent pixels around a certain range in either the x-axis direction or the y-axis direction, Alternatively, a value obtained by combining them corresponds to a binary image in which the value of a pixel that is equal to or greater than a predetermined threshold is 1 and the value of other pixels is 0.

  For example, a Canny edge detector or a Hough converter is used for edge detection. However, the method used for edge detection is arbitrary, and is not limited to these detectors and converters. FIG. 6 shows an example of the first edge image. In the case of FIG. 6, a plurality of edges run substantially in parallel in the vertical direction in the center of the image. These edges are due to the gap between the display surface 102 and the background plate 105, the boundary of the electronic blackboard housing (board 101) that supports the background plate 105 and its structure from the back side, the reflection of the background plate 105 on the display surface 102, and the like. It has occurred.

  In the first edge image shown in FIG. 6, the two edges extending obliquely in the left half region are generated by the boundary between the electronic blackboard housing (board 101) and the display surface 102. The rectangular area in the center of the first edge image is an area obtained by imaging the display surface 102 on which white light is projected, so no edge is extracted. If an object other than the floor is reflected on the right side of the image, various edges are extracted in this area.

  In step 303, the background board area extraction function unit (CPU 201) combines the parallel edges included in the first edge image into one by image processing so as to be handled as an edge synthesis area. That is, edge composition and removal are executed. At this time, the background board area extraction function unit (CPU 201) operates as an edge synthesis removed image generation unit.

  Here, for the synthesis and removal of edges, morphological conversion processing (expansion processing, contraction processing, or a combination thereof) that fills the gaps between edges, flood fill processing that specifies an area surrounded by edges, and the like are used. . However, these are merely examples, and any other method can be used.

  Further, the background plate area extraction function unit (CPU 201) removes an edge extending in the negative direction of the x axis from the coordinates of the display surface instruction target in the first edge image. Thereby, an edge generated at the boundary between the electronic blackboard housing (board 101) and the display surface 102 can be removed. FIG. 7 shows an example of an image (edge synthesis / removed image) after the edge synthesis processing and removal processing are executed. In FIG. 7, a gray colored region at the center of the screen is a region obtained by edge synthesis processing. In the case of FIG. 7, only the area corresponding to the background plate 105 remains. However, when an object other than the floor, such as a person, an object, a wall, or a window, is reflected in the entire image, the edge composition area generated from these remains in the right half of the edge composition removal image even at this time. .

  Therefore, in step 304, the background board area extraction function unit (CPU 201) executes a process of extracting only the boundary line corresponding to the background board area from the edge synthesis removed image. Here, the background board area extraction function unit (CPU 201) operates as a second edge image generation part and a background board area extraction part.

  First, the function as the second edge image generation unit will be described. At this time, the background board area extraction function unit (CPU 201) applies the edge detection process similar to that in step 302, and creates an image of the edge component (second edge image) included in the edge synthesis area. That is, only edges having a large change amount are extracted from the edge synthesis removed image.

  Next, the function as a background board area extraction unit will be described. At this time, the background plate area extraction function unit (CPU 201) utilizes the fact that the background plate 105 exists on the right side (the side with the larger x-axis coordinate) than the coordinates of the display surface instruction target, Only the edge corresponding to 105 is extracted efficiently. That is, based on the relative positional relationship between the coordinates of the display surface instruction target and the edges corresponding to the background plate 105, only the edges corresponding to the background plate 105 are extracted.

  Specifically, an edge that meets first and an edge that meets second when the coordinate of the display surface instruction target in the second edge image starts as the starting point in the positive direction of the x-axis are displayed in the region corresponding to the background plate 105. The left and right border lines. Further, an area sandwiched between these two boundary lines and the upper and lower ends of the second edge image is regarded as a background board area.

  By applying this edge selection method, even when a person, an object, a wall, a window, or the like is reflected in the imaging area of the floor 106 and the corresponding edge synthesis area remains in the right half of the edge synthesis removal image, the background plate Only the border of the region can be selected reliably.

  Note that the boundary line of the actual edge composition region is generally formed by combining a plurality of edges. That is, the boundary line is often not a simple straight line. Specifically, the boundary line is often configured as a collection of a plurality of line segments. In this case, a method of extracting the first meeting edge and the second meeting edge when advancing radially from one display surface pointing target in the positive direction of the x-axis, so as to be parallel to each side of the background plate 105 It is necessary to divide an imaginary line that connects two display surface indication targets to be placed, and to extract the edge that meets first and the edge that meets second when going in the positive x-axis direction from the dividing point It becomes. However, these methods require a large amount of calculation for image processing when there are many line segments constituting the edge. Note that, due to the influence of lens distortion or the like, the line segment constituting the boundary line may be an arcuate arc instead of a straight line.

  Therefore, the present inventor proposes the following processing contents as a method for efficiently detecting the edge that gives the boundary line of the background plate area. Specifically, a method for approximating an edge composed of a large number of line segments to a function and realizing an efficient image processing is proposed. Here, a case where the boundary line is approximated to a straight line will be described as an example of the approximation method.

  FIG. 8 shows an example of the second edge image. In the case of FIG. 8, the two vertical lines in the center of the image are the boundary lines constituting the edge synthesis area. In FIG. 8, for the sake of simplicity, only the boundary line corresponding to the background plate 105 is shown. However, when there is reflection of an object other than the floor, the other boundary is located on the right half side of the second edge image. As already mentioned, there is also a line.

  When the linear approximation method is applied, the background plate area extraction function unit (CPU 201) divides the line segment connecting the coordinates of the display surface instruction target 2 and the display surface instruction target 4 into equal parts on the second edge image. Calculate the coordinates of the points. Next, the background plate area extraction function unit (CPU 201) proceeds from each division point in the positive direction of the x-axis (right direction in FIG. 8), and the coordinates of the intersection with the first meeting edge and the second meeting edge. Detect the coordinates of the intersection of Next, the background plate area extraction function unit (CPU 201) collects only the coordinates of the intersection with the edge that meets first, and obtains an approximate straight line that passes through those coordinates by the method of least squares. For example, the boundary line of the lower left background plate region in FIG. 8 is obtained as a linear expression given by y = ax + b. Similarly, if only the coordinates of the intersection point with the edge that meets the second are collected and an approximate straight line passing through those coordinates is obtained by the least square method, a boundary line on the lower right side can be obtained.

  When the same procedure is executed for the display surface instruction target 3 and the display surface instruction target 4, the boundary line on the upper left side and the upper right side can be obtained. Note that any number of equal numbers of line segments connecting the display surface instruction targets may be used, and the number is not limited to that illustrated in FIG. Further, the boundary line approximation method may be a method other than the least square method mentioned here, or may be approximated to a function other than a linear expression.

  Note that if the width of the edge composite area included in the second edge image changes greatly in the middle, or the edge composite area is divided, the coordinates of the intersection used for approximation are greatly shifted from other coordinates. Even in this case, if the difference from each intersection is verified after the boundary line is approximated and the boundary line is re-approximed after excluding coordinates that are greatly shifted, the background plate area close to the contour of the original background plate 105 is obtained. A boundary line can be obtained.

  By the way, the background board area extracted by the above-described method may be wider than the original background board area due to reflection of the background board 105 on the display surface 102 or the like. When such a detection result is assumed, a fraction of the width of the background board area extracted by the above-described method may be scraped from both sides or one side and used as the final background board area.

  In addition, since the image processing efficiency is better when the width of the background plate area is constant, only the first edge encountered in the x-axis direction from the display surface instruction target is extracted, and the edge or the more approximate boundary line is extracted. An area having a certain width in the positive direction of the x axis may be regarded as a background board area.

  Similarly, only the edge that has been encountered second in the positive direction of the x-axis from the display surface indication target is extracted, and an area having a certain width in the negative direction of the x-axis from the edge or a further approximate boundary line is extracted. It may be regarded as a background board area.

(Coordinate detection function)
When an area (background board area) corresponding to the background board 105 is extracted from the imaging screens of the camera A 103 and the camera B 104 by executing the background board area extraction function described above, the coordinates on the screen indicated by the coordinate indicator are displayed. Are detected using the principle of triangulation. Here, the outline of the coordinate detection process in the present embodiment will be described.

  First, the CPU 201 cuts out an image of the background plate area determined by the above-described background plate area extraction unit as a reference image from the entire image captured by the camera A 103 and the camera B 104 in a state where there is no coordinate pointing object on the detection surface. . At this time, the CPU 201 functions as a reference image storage control unit. The reference image corresponding to the camera A 103 is stored in the reference image storage area of the RAM 203 as the left reference image, and the reference image corresponding to the camera B 104 is stored as the right reference image. Information giving the background board area is stored in the RAM 203.

  Next, the CPU 201 updates the image of the background plate area determined by the above-described background plate area extraction unit from the entire image captured by the camera A 103 and the camera B 104 with the coordinate indicator inserted on the detection surface. Cut out as. At this time, the CPU 201 functions as a coordinate pointing object image storage. The latest image corresponding to the camera A103 is stored as a left coordinate indicator image, and the latest image corresponding to the camera B104 is stored as a right coordinate indicator image in the coordinate indicator image area of the RAM 203.

  Next, the CPU 201 extracts only the coordinate indicator as a subject from the difference image between each coordinate indicator image and the corresponding reference image. That is, the CPU 201 functions as a subject extraction unit.

  Subsequently, the CPU 201 detects (calculates) coordinates on the display surface of the coordinate indicator based on the positional information of the extracted subject in the pair of left and right coordinate indicator images. That is, the CPU 201 functions as a coordinate detection unit. At this time, since the pair of left and right coordinate indicator images include only the coordinate indicator image that is the subject, it is possible to always detect (calculate) accurate coordinates.

(Summary)
As described above, in the electronic blackboard system according to the embodiment, the background board area extraction process is executed before the electronic blackboard 100 is used. As a result, only the background plate region corresponding to the background plate 105 can be automatically extracted from the entire image captured by the camera A 103 and the camera B 104. If the extracted background board area or the background board area shaped into a rectangle by further image processing is used, when the electronic blackboard enclosure (board 101) is deformed due to temperature change or aging change, the camera mounting position is slightly shifted. In this case, the background plate region can be automatically extracted so that the coordinates of the coordinate pointing object can be accurately detected without fine adjustment of attachment of the camera or the like by the user or the like.

  In addition, by incorporating the automatic extraction function of the background plate area, accurate coordinate detection can always be realized with less burden than in the past.

  DESCRIPTION OF SYMBOLS 100 ... Electronic blackboard, 101 ... Board, 102 ... Display surface, 103 ... Camera A, 104 ... Camera B, 105 ... Background board, 106 ... Floor, 200 ... PC, 201 ... CPU, 202 ... ROM, 203 ... RAM, 204 ... HDD, 205 ... Video card, 206 ... Display I / F, 207 ... External I / F, 208 ... Internal bus, 209 ... Display cable, 210 ... Projector, 211 ... Connection cable.

Claims (12)

Display surface designation screen display control means for displaying a display surface designation screen including one or more display surface indication targets for instructing the contact position of the coordinate indicator on the display surface having a predetermined background plate on the outer periphery;
One or more imaging means for imaging a coordinate indicator inserted on a detection surface near the sky above the display surface;
From the reference image captured by the imaging means when no coordinate indicator is inserted on the detection surface and the target image captured by the imaging means when the coordinate indicator is inserted on the detection surface, Target coordinate detection means for detecting target coordinates corresponding to the display surface instruction target in the captured image of the imaging means;
First edge image generation means for generating a first edge image obtained by extracting an edge of the reference image;
In the first edge image, an edge synthesis region is generated by synthesizing adjacent parallel edges, and an edge synthesis removal image is generated by removing an edge that does not satisfy the first relative positional relationship with the target coordinate. Edge synthesis removal image generating means for performing,
Second edge image generation means for generating a second edge image obtained by extracting an edge of the edge synthesis removed image;
Background plate area extracting means for extracting, in the second edge image, an edge satisfying a second relative positional relationship with respect to at least one of the display surface instruction target coordinates as a background plate area corresponding to the outline of the background plate; ,
An electronic blackboard system comprising: The electronic blackboard system according to claim 1,
A reference image storage control means for storing a reference image obtained by cutting out an area extracted by the background plate area extraction means from an image taken by a plurality of the imaging means in a state where there is no coordinate indicator on the detection surface;
Coordinate indicator image storage for storing an image of a coordinate indicator obtained by cutting out an area extracted by the background plate area extracting means from images picked up by a plurality of the imaging means in a state where the coordinate indicator is inserted on the detection surface. Control means;
Subject extraction means for extracting only the subject that hits the coordinate indicator from the coordinate indicator image based on the reference image;
Coordinate detection means for detecting the coordinates of the coordinate indicator from the position in the coordinate indicator image of the subject extracted by the subject extraction means;
An electronic blackboard system comprising: The electronic blackboard system according to claim 1,
The electronic blackboard system characterized in that the edge satisfying the second relative positional relationship is an edge where a virtual line extending in a direction in which a background plate exists starting from the target coordinates meets a second edge and a second edge. . The electronic blackboard system according to claim 1,
The edge satisfying the second relative positional relationship is an edge where a virtual line extending in a direction in which a background plate exists starting from the target coordinate first meets,
The backgroundboard area is an area within a range in a direction away from the target coordinates by a certain width from the first encountering edge. The electronic blackboard system according to claim 1,
The edge satisfying the second relative positional relationship is an edge where a virtual line extending in a direction in which a background plate exists starting from the target coordinates meets second.
The background board area is an area within a range in a direction of approaching the target coordinates by a certain width from the second edge to be encountered. The electronic blackboard system according to claim 1,
The edge satisfying the second relative positional relationship is an edge where a virtual line extending in a direction in which a background plate exists starting from a dividing point on a line segment connecting the target coordinates and the plurality of target coordinates is first encountered. An electronic blackboard system, characterized in that an approximate function for a set of intersection points and an approximation function for a set of intersection points of the virtual line and the edge where the virtual line meets secondly. The computers that make up the electronic blackboard system
Display surface designation screen display control means for displaying a display surface designation screen including one or more display surface indication targets for instructing the contact position of the coordinate indicator on the display surface having a predetermined background plate on the outer periphery;
Control means for controlling one or more imaging means for imaging a coordinate indicator inserted on a detection surface near the sky above the display surface;
From the reference image captured by the imaging means when no coordinate indicator is inserted on the detection surface and the target image captured by the imaging means when the coordinate indicator is inserted on the detection surface, Target coordinate detection means for detecting target coordinates corresponding to the display surface instruction target in the captured image of the imaging means;
First edge image generation means for generating a first edge image obtained by extracting an edge of the reference image;
In the first edge image, an edge synthesis region is generated by synthesizing adjacent parallel edges, and an edge synthesis removal image is generated by removing an edge that does not satisfy the first relative positional relationship with the target coordinate. Edge synthesis removal image generating means for performing,
Second edge image generation means for generating a second edge image obtained by extracting an edge of the edge synthesis removed image;
In the second edge image, a background board area extracting means for extracting an edge satisfying a second relative positional relationship with respect to the target coordinates as a background board area corresponding to the outline of the background board;
A program characterized by functioning as The program according to claim 7 is a computer,
A reference image storage control means for storing a reference image obtained by cutting out an area extracted by the background plate area extraction means from an image taken by a plurality of the imaging means in a state where there is no coordinate indicator on the detection surface;
Coordinate indicator image storage for storing an image of a coordinate indicator obtained by cutting out an area extracted by the background plate area extracting means from images picked up by a plurality of the imaging means in a state where the coordinate indicator is inserted on the detection surface. Control means;
Subject extraction means for extracting only the subject that hits the coordinate indicator from the coordinate indicator image based on the reference image;
Coordinate detection means for detecting the coordinates of the coordinate indicator from the position in the coordinate indicator image of the subject extracted by the subject extraction means;
A program characterized by functioning. The program according to claim 7,
The edge satisfying the second relative positional relationship is an edge where a virtual line extending in a direction in which a background plate exists with the target coordinate as a starting point and an edge where the imaginary line meets first and second. The program according to claim 7,
The edge satisfying the second relative positional relationship is an edge where a virtual line extending in a direction in which a background plate exists starting from the target coordinate first meets,
The background plate region is a region in a range in a direction away from the target coordinates by a certain width from the first meeting edge. The program according to claim 7,
The edge satisfying the second relative positional relationship is an edge where a virtual line extending in a direction in which a background plate exists starting from the target coordinates meets second.
The background plate area is an area that is within a range in a direction of approaching the target coordinates by a certain width from the second meeting edge. The program according to claim 7,
The edge satisfying the second relative positional relationship is an edge where a virtual line extending in a direction in which a background plate exists starting from a dividing point on a line segment connecting the target coordinates and the plurality of target coordinates is first encountered. A program characterized in that an approximate function for a set of intersection points and an approximate function for a set of intersection points between the virtual line and the edge where the virtual line meets secondly are given.

Images