JP2009026262A - Display image acquisition device and display image acquisition program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a display image acquisition device and a display image acquisition program that calculate, on the basis of image display signals, the position of a pointer formed integrally with a display image. <P>SOLUTION: The display image acquisition device has: a video capturing device 42 for obtaining image display signals; an image sampling device 43 for extracting the image display signals continuously in a fixed cycle; an image conversion device 44 for converting the image display signals into display images according to the image display signals; and a pointer position calculating device 45 for creating a difference image from successive display images, binarizing the difference image, specifying an area where a difference value exists as a pointer candidate area occupied by a pointer, and causing a database 48 to store a portion of the pointer candidate area that is closest from a reference set at the left top of the difference image. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a display image acquisition apparatus that acquires and accumulates display images displayed on a display, and more particularly to a display image acquisition apparatus that calculates the position of a pointer integrally formed with a display image.

  Use an information terminal such as a PC (Personal Computer) or PDA (Personal Digital Assistant) to view a presentation image or sound provided from a remote location, or save the presentation and view it later. It is known. For example, Patent Document 1 discloses saving an image for presentation as video information.

  In general, if there is a slide image that is a still image, audio for explaining the slide image, and information on the position and shape of a pointer indicating a region of interest in the slide image (hereinafter referred to as pointer information), the presentation is a presentation. It is thought that the contents of can be fully communicated. In view of such circumstances, for example, Patent Document 2 discloses a recording apparatus that acquires and records a slide image, sound, and pointer information directly from an information terminal.

  Furthermore, as a pointer information acquisition technique, as disclosed in, for example, Patent Document 3 and Patent Document 4, pointer information using a pointer or a laser pointer, or pointer information acquired by a user's hand gesture is known.

  On the other hand, a display image acquisition device that acquires a display signal (hereinafter referred to as an image display signal) such as an RGB signal output from the information terminal, converts the display signal into a display image corresponding to the image display signal, and stores the converted display image. Is also known. The display image acquisition device monitors the image display signal and acquires the image display signal when the signal changes greatly. For example, the display image acquisition device can discriminate switching of a display image such as a slide image, and accumulates the display image efficiently. it can.

JP 2000-187542 A JP 2003-58901 A JP 2000-276297 A Japanese Patent Laid-Open No. 2001-5609

  By the way, since the storage disclosed in the above-mentioned Patent Document 1 is video information, a large-capacity storage device is required, which may increase the equipment cost. In addition, when using the recording device disclosed in Patent Document 2, there is a limitation that software for using the recording device must be installed in advance in the information terminal. Furthermore, the pointer position acquisition techniques disclosed in Patent Document 3 and Patent Document 4 are not intended for image display signals.

  The present invention has been made in view of such circumstances, and provides a display image acquisition device and a display image acquisition program for calculating the position of a pointer integrally formed on a display image from an image display signal. Objective.

  According to the first aspect of the present invention, the acquisition means for acquiring the image display signal, the extraction means for continuously extracting the image display signal at a constant period, and the image display signal are set according to the image display signal. A conversion means for converting into a display image, a difference image is generated from the continuous display image and binarized, an area where the difference value exists is specified as a pointer candidate area occupied by the pointer, and the difference image is included in the pointer candidate areas And a pointer position calculation means for storing a part of the pointer candidate area closest to the reference in the storage means as pointer position information when the upper left of the reference is used as a reference. .

  The invention according to claim 2 is characterized in that, in the invention according to claim 1, the pointer position calculation means stores position information of each pointer at each time when there are two pointer candidate areas. Yes.

  According to a third aspect of the present invention, in the first or second aspect of the invention, the pointer position calculation means is configured when the number of difference values obtained as a result of binarization is greater than twice the number of pixels occupied by the pointer. The processing for calculating the position of the pointer at that time is stopped.

  The invention according to claim 4 is the invention according to any one of claims 1 to 3, wherein the pointer position calculation means calculates the position of the pointer at that time when there are three or more pointer candidate areas. It is characterized by canceling the processing.

  According to a fifth aspect of the present invention, in the invention according to any one of the first to fourth aspects, the pointer position calculation means performs an edge extraction process for each of the consecutive pointer candidate areas, and the extraction process When the similarity between the edge shapes is high, the position of the pointer is calculated.

  The invention according to claim 6 is the invention according to any one of claims 1 to 5, wherein the pointer position calculation means shapes the edge shape and stores the pointer shape. The display image acquisition device according to any one of 1 to 5.

  According to a seventh aspect of the present invention, there is provided a computer comprising: an acquisition unit that acquires an image display signal; an extraction unit that continuously extracts an image display signal at a constant period; and the image display signal. Conversion means for converting into a display image in accordance with the signal, a difference image is generated from the continuous display image, binarized, an area where the difference value exists is specified as a pointer candidate area occupied by the pointer, When the upper left of the difference image is used as a reference, a display image acquisition program that functions as pointer position calculation means for storing a part of a pointer candidate area closest to the reference as position information of the pointer in the storage means It is.

  According to the first aspect of the present invention, the position of the pointer integrally formed on the display image can be calculated from the image display signal.

  According to the second aspect of the present invention, the movement of the pointer can be grasped in time series.

  According to the third aspect of the present invention, it is possible to suppress erroneous determination that the switching of the slide image is the movement of the pointer.

  According to the fourth aspect of the present invention, it is possible to suppress erroneous determination using an image corresponding to the size of the pointer as a pointer.

  According to the fifth aspect of the present invention, it is possible to suppress suppression of erroneous determination using an image other than the pointer as a pointer.

  According to the invention described in claim 6, the reproducibility of the pointer is improved.

  According to the seventh aspect of the present invention, the position of the pointer integrally formed on the display image can be calculated from the image display signal.

DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, exemplary embodiments of the invention will be described with reference to the drawings.
FIG. 1 is a configuration diagram of a system (hereinafter referred to as a display image acquisition system) to which a display image acquisition device according to an embodiment of the present invention is applied, and FIG. 2 is a block diagram showing a hardware configuration of the display image acquisition device.

  As shown in FIG. 1, the display image acquisition system includes an information terminal 10, a splitter 202, a projector 30, a display image acquisition device 40, and the like.

  The information terminal 10 includes an input device such as a keyboard and a mouse, a display device such as a liquid crystal display and a CRT display, and a control device such as a computer that controls these devices. In FIG. 1, a notebook PC is shown, but a mobile phone, a PDA, or a stand-alone PC may be used. The control device outputs an image to be displayed on the display device as an image display signal such as a VGA signal or an RGB signal, and outputs the signal to the splitter 20.

  The splitter 20 is a device that separates an image display signal, and is installed between the information terminal 10 and the projection device 30. Upon reception of the image display signal, the splitter 20 separates and outputs one to the projection device 30 and the other to the display image acquisition device 40 connected to the splitter 20 by a cable or the like. In FIG. 1, the splitter is separated in two directions, but may be in three or more directions.

  When the projection device 30 is a so-called projector and receives an image display signal, the projection device 30 projects an image corresponding to the signal onto a screen, a wall surface, or the like as a projection image. Instead of the projection device 30, the display device described above may be used.

  The display image acquisition device 40 is a device that receives an image display signal, converts it into a display image corresponding to the image display signal, and accumulates the display image in a storage device. The display image acquisition device 40 is a computer, that is, as shown in FIG. 2, a processing device such as a CPU 40a, SRAM (Static Random Access Memory), DRAM (Dynamic RAM), SDRAM (Synchronous DRAM), NVRAM (Non Volatile RAM). Hardware configuration in which a RAM 40b such as a flash memory, a ROM (Read Only Memory) 40c such as a flash memory, an I / F 40d that inputs and outputs with an external device such as a splitter 20, and a magnetic disk such as a hard disk (not shown) are connected by a bus 40e. It is realized by.

  Therefore, each function in the display image acquisition device 40 is realized by the CPU 40a reading a required program stored in the ROM 40c, the hard disk, or the like and performing a calculation according to the program. The control device of the information terminal 10 has the same configuration. In addition, as such a program, it can be set as the program according to the flowchart mentioned later.

Here, the display image acquisition device 40 described above will be described in more detail with reference to FIG.
FIG. 3 is a functional block diagram showing a main configuration of the display image acquisition device 40.
As shown in FIG. 3, the display image acquisition device 40 includes an input / output device 41, a video capture device 42, an image sampling device 43, an image conversion device 44, a pointer position calculation device 45, an update device 46, a text extraction device 47, It consists of a database 48 and a bus 49. Each device is connected to each other via a bus 49.

  The input / output device 41 includes a connector for connecting a cable or the like. The input / output device 41 is connected to the above-described splitter 20 via a cable, receives an image display signal, and outputs it to the video capture device 42. In addition, it is good also as a wireless form, without using a cable.

The video capture device 42 is an example of an acquisition unit of the present invention, and captures an image display signal output from the input / output device 41.
The image sampling device 43 is an example of the extracting means of the present invention, and continuously extracts the image display signal received from the video capturing device 42 at a constant period. The fixed period may be, for example, a video rate every 1 / 29.97 seconds or about every 0.5 seconds. The shorter this interval, the more accurately the switching of slide images and the detection of pointer movement can be performed.

  The image conversion device 44 is an example of the conversion means of the present invention, acquires the image display signal extracted by the image sampling device 43, and sequentially converts it into a display image corresponding to the image display signal. As the display image, for example, various formats such as a BMP (bitmap) format, a JPEG format, a TIFF format, and a GIF format can be used. After the conversion to the display image, the image conversion device 44 sequentially transmits the display image to the pointer position calculation device 45.

  The pointer position calculation device 45 is an example of the pointer position calculation means of the present invention, generates a difference image from continuous display images, binarizes it, and specifies an area where the difference value exists as a pointer candidate area occupied by the pointer Then, in the pointer candidate area, when the upper left of the difference image is used as a reference, a part of the pointer candidate area closest to the reference is stored in the database 48 as pointer position information. Focusing on the fact that there is symmetry as a characteristic of the shape of the pointer, a region having the target property among the regions where the difference value exists may be used as the pointer candidate region. This increases the accuracy of the determination of being a pointer.

The update device 46 assigns title information, a presenter name, and the like as attribute information to all display images classified as the same presentation.
The text extraction device 47 extracts text information by applying a character recognition process (OCR) to the display image selected as the storage target. In this process, the display image recorded and accumulated based on the text information is searched, and the name of the presenter and the title (title) of the presenter described in the display image are assigned as an index to the classified display image group. To do.

  The database 48 stores the display image converted by the image conversion device 44 and the pointer position information calculated by the pointer position calculation device 45 for each time.

Next, the operation of the pointer position calculation device 45 described above will be described with reference to FIGS.
4 is a flowchart showing an example of the operation of the pointer position calculation device 45, FIG. 5 is a diagram for explaining generation of a difference image, FIG. 6 is a graph of a binarized curve, and FIG. 7 is a calculation of the position of the pointer. FIG. 8 is a diagram for explaining the case where the calculation of the pointer position is stopped, FIG. 9 is another diagram for explaining the generation of the difference image, and FIG. 10 is the calculation of the pointer position. FIG. 11 is a formula for determining the similarity of the shape before and after the movement of the pointer, and FIG. 12 is another diagram for explaining the case where the calculation of the position of the pointer is stopped. FIG. 13 shows an example of storage of pointer positions in the database.

As shown in FIG. 4, the pointer position calculation device 45 first receives display images from the image conversion device 44 sequentially, and generates a difference image from successive display images (step S1). For example, as shown in FIG. 5 (a), after receiving the display image I _t-1 at time t-1, as shown in FIG. (B), when receiving the display image I _t at time t from the display image _{I t} display image _{I t-1} Tokyo, as shown in FIG. (c), to generate a difference image _{D t.} Thus, the pointer Pt before and after movement is generated in the difference image D _t.

Pointer position calculation means 45 then binarizes the difference image _{D t} (step S2). For example, binarization is performed by a binarization curve L shown in FIG. Binarization curve L has a large difference in density value of the display image I _t and the display image I _t-1 as the distance from 0, the density value of the display image I _t and the display image I _t-1 closer to 0 This means that the difference between is small.

Accordingly, a portion where the difference in density value is clear is set as a difference value 1, and a portion where the density value is not clear is set as a difference value 0. The threshold value P that divides this clarity is not specifically shown as a numerical value in FIG. As a result, the difference image D _t, as shown in FIG. 7 (a), part a difference value the difference value is 0 is divided into a portion to be the 1.

Pointer position calculation means 45 determines where the difference value from the entire difference image D _t is counted the number of parts is 1, whether the number is 20 to 1,000 (step S3). That is, the count if the result is greater than 1000, as shown in FIG. 8, instead of the display and move the pointer Pt, for example, the display image I _t-1 from the difference image D _t by the switching of the display image I _t Assuming that a large number of portions having a difference value of 1 appear and are counted, the calculation processing of the pointer position at that time is stopped, and the processing at the next time is performed. On the other hand, if the counting result is less than 20, the calculation processing of the pointer position at that time is stopped because the pointer is not displayed or moved, and the processing at the next time is performed.

Note that the criterion 20 to 1000 is not particularly limited as long as it is within the range of the number of portions occupied by the pointer in the display image and multiples thereof, and is changed according to the size of the pointer. Also good. This is because when only the movement of the pointer Pt in the difference image D _t remains as a difference is because many two pointers Pt even appears on the difference image D _t. As a result, when it is determined that the counting result is 20 or more and 1000 or less, the process proceeds to the subsequent process.

Pointer position calculation means 45 then determines the locality of the difference image _{D t} (step S5). More specifically, as shown in FIG. 7 (b), the upper left corner of the difference image D _t as a reference point from O, left to right, from the top, searching for a portion where the difference value is 1. And when the said part is detected, it searches up and down, right and left on the basis of the part, and the part from which a difference value is set to 1 is counted. In addition, a counted flag or the like is assigned to the portion once counted so that duplicate counting is not performed. As a result, when the portion having the difference value of 1 forms one aggregate (or area), the aggregate is set as a pointer candidate area AR as shown in FIG.

Search of the portion where the difference value is 1 is carried out up to the lower right corner of the difference image D _t. As a result, when the pointer candidate area AR is one or two, the subsequent processing is performed. For example, as shown in FIG. 9, the pointer Pt has moved, but when the amount of movement is small and the difference image _Dt is generated, the pointer Pt overlaps before and after the movement. Show.

On the other hand, when the number of pointer candidate areas AR is zero or three or more, the calculation processing of the pointer position at that time is stopped. The case where the pointer candidate area is three, for example, as shown in FIG. 10, the pointer Pt is included two in the difference image D _t, the animation effects like slide image, the size of the pointer Pt in some cases such as character information Tx, image information, etc. it is included in the difference image D _t which corresponds to the.

  Next, the pointer position calculating device 45 determines whether or not the shapes (outlines) of the pointer Pt before and after the movement match (step S5). It should be noted that whether or not the shapes of the pointer Pt before and after the movement match is not a perfect match, and it is sufficient if the similarity is generally high.

Here, whether or not the degree of similarity before and after the movement of the pointer Pt is high is determined by the mathematical formula shown in FIG. In the equation shown in FIG. 11, E _t (x, y) is an edge map of the pointer candidate area AR at time t, and E _t-1 (x + Δx, y + Δy) is an edge map of the pointer candidate area AR at time t−1. Is shown. The edge map of the pointer candidate area AR can be generated by a general technique, and can be generated by combining, for example, a Sobel filter and binarization processing.

The calculation result S according to the mathematical formula shown in FIG. 11 is _{1 when} the edge shape in the edge map E _t and the edge map E _t−1 is the same except for the movement in the pointer candidate area AR, and is very different when it is completely different. It becomes a small value. For example, if such a difference image D _t as shown in FIG. 12 is obtained, it is determined that a completely different shape, a very small value. When the pointer position calculation device 450 determines that the positions before and after the movement of the pointer Pt are approximately the same, the pointer position calculation device 450 performs subsequent processing. Conversely, if the calculation result is a very small value, it is determined that the degree of similarity is low, and the pointer position calculation process at that time is stopped.

Next, the pointer position calculation device 45 calculates the position of the pointer Pt (step S6). In the calculation of the position of the pointer Pt, as shown in FIG. 7 (b), for example, in the case where the upper left corner of the difference image D _t as a reference point O, closest to the reference point O, in the pointer candidate area AR It can be a part where the difference value is 1. At this time, when the pointer Pt at time t and the pointer Pt at time t−1 coexist in the difference image D _t , the respective positions are calculated for each time. As a result, as shown in FIG. 6B, the coordinate A of the pointer Pt at time t-1 can be (25, 10), and the coordinate B of the pointer Pt at time t is (60, 50). can do.

  Next, the pointer position calculation device 45 causes the database 48 to store the position of the pointer Pt calculated by the process of step S6. As shown in FIG. 13, it is desirable to store the data in the database 48 at each time, thereby enabling time-series tracking of the pointer Pt. Further, the shape of the pointer Pt may also be stored. The shape can be realized by storing the edge of the pointer Pt obtained by the edge extraction process in step S5 described above. Further, there may be a case where the position of one pointer before the sampling position is already stored. In such a case, an appropriate value such as an average value or the latest stored value may be used from the positions of a plurality of pointers stored for the same time.

Incidentally, as described with reference to FIG. 9, when the pointer Pt is contained overlapping in the difference image D _t, as shown in FIG. 14, the pointer candidate area AR is one. In this case, even if the edge extraction process is performed on each of the pointers Pt and the mathematical formula shown in FIG. Accordingly, in such a case, it is determined that the two pointers Pt are the same even if the calculation result is approximately 0.5 or more. Otherwise, it is determined that the pointer Pt is the text information Tx or the like. To do. Thereby, the position of the pointer can be detected even if the pointer Pt is moved by a small amount. According to FIG. 14, as a result, the obtained coordinates C at time t-1 are, for example, C (25, 10).

  Further, the display image acquisition device 40 may be provided with an information transmission device that transmits display images and pointer information stored in the database 48 in response to a request from the information terminal 10. The information terminal 10 can display the received display image with the pointer Pt superimposed at the time and place according to the pointer information.

  Further, the pointer Pt may be displayed by interpolating between the times based on the pointer information. FIG. 15 is a diagram showing the locus of the pointer Pt from time t-2 to time t-1. The pointer information includes the positions of the pointers Pt at the respective times t-2, t-1, and t + 1. When these are displayed on the display image, the pointers Pt are displayed discontinuously. It will be. Therefore, it is possible to continuously move the pointer Pt by using a method such as an interpolation method based on each time. For example, spline interpolation or the like may be used as the interpolation method. The more the time of the stored pointer information, the smoother and more accurate pointer position can be reproduced.

  Furthermore, the display image after the pointer Pt is extracted and sharing many parts other than the pointer may be collected and stored in the database 48 as one display image. By storing the display image and the pointer separately, the storage capacity can be reduced.

  The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific embodiments according to the present invention, and various modifications can be made within the scope of the gist of the present invention described in the claims. Deformation / change is possible. For example, the program of the present invention can be provided not only by communication means but also stored in a recording medium such as a CD-ROM.

  In addition, any name such as a cursor, a mouse pointer, or a mouse cursor can be applied to the pointer in the above description. Further, in the above description, the portion where the difference value is 1 can be a single pixel in the display device. For example, an aggregate of four or nine pixels can be a portion where the difference value is 1. . Furthermore, the reference points for calculating the position of the pointer may be any of the four corners of the difference image.

  According to the present invention, the position of the pointer integrally formed on the display image can be calculated from the image display signal, and the industrial applicability is high.

It is a block diagram of a display image system. It is a block diagram which shows the hardware constitutions of a display image acquisition apparatus. It is a functional block diagram which shows the principal part structure of a display image acquisition apparatus. It is a flowchart which shows an example of operation | movement of a display image acquisition apparatus. It is a figure for demonstrating the production | generation of a difference image. It is a graph of a binarization curve. It is a figure for demonstrating calculation of the position of a pointer. It is a figure for demonstrating the case where calculation of the position of a pointer is stopped. It is another figure for demonstrating the production | generation of a difference image. It is another figure for demonstrating the case where the calculation of the position of a pointer is stopped. It is a mathematical formula for determining the similarity of the shape before and after the movement of the pointer. It is another figure for demonstrating the case where the calculation of the position of a pointer is stopped. It is the example of a memory | storage of the position of the pointer in a database. It is another figure for demonstrating calculation of the position of a pointer. It is another figure for demonstrating calculation of the position of a pointer.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Information terminal 20 Splitter 30 Projector 40 Display image acquisition apparatus 40a CPU
40b RAM
40c ROM
40d I / F
400e bus 41 input output device 42 a video capture device 43 images the sampling device 44 the image converter 45 pointer position calculating unit 46 updates device 47 text extractor 48 database 49 bus _{I t,} I _t-1 display image D _t the difference image Pt pointer AR pointer candidate area Tx text information

Claims (7)

Obtaining means for obtaining an image display signal;
Extracting means for continuously extracting the image display signal at a constant period;
Conversion means for converting the image display signal into a display image corresponding to the image display signal;
When the difference image is generated from the continuous display image and binarized, an area where the difference value exists is specified as a pointer candidate area occupied by the pointer, and the upper left of the difference image is used as a reference among the pointer candidate areas Pointer position calculation means for storing a part of the pointer candidate area closest to the reference in the storage means as pointer position information;
A display image acquisition apparatus comprising:   The display image acquisition apparatus according to claim 1, wherein the pointer position calculation unit stores position information of each pointer at each time when there are two pointer candidate areas.   The pointer position calculation means, when the number of difference values obtained as a result of binarization is larger than twice the number of pixels occupied by the pointer, stops the processing for calculating the position of the pointer at that time. The display image acquisition device according to claim 1.   The said pointer position calculation means stops the process which calculates the position of the pointer at the time, when the said pointer candidate area | region is three or more places, The any one of Claim 1 to 3 characterized by the above-mentioned. Display image acquisition device.   The pointer position calculation means performs an edge extraction process for each of the consecutive pointer candidate areas, and stops the process of calculating the pointer position when the similarity of the edge shapes by the extraction process is low The display image acquisition device according to claim 1, wherein:   The display image acquisition apparatus according to claim 1, wherein the pointer position calculation unit shapes the edge shape and stores the shape of the pointer. Computer
Obtaining means for obtaining an image display signal;
Extracting means for continuously extracting the image display signal at a constant period;
Conversion means for converting the image display signal into a display image corresponding to the image display signal;
When the difference image is generated from the continuous display image and binarized, an area where the difference value exists is specified as a pointer candidate area occupied by the pointer, and the upper left of the difference image is used as a reference among the pointer candidate areas A display image acquisition program for causing a part of a pointer candidate area closest to the reference to be stored in a storage unit as pointer position information.

Images