JP7331578B2 - Display device, image display method, program - Google Patents

Description

The present invention relates to a display device, an image display method, and a program.

Techniques for saving a copy of part or all of a screen displayed on a visual output device such as a computer display or a saved image (hereinafter referred to as first image) are known. The first image may be called a screenshot, captured image, or the like. The user can display the saved first image on a display or the like when necessary. Therefore, the user can easily save the screen that he/she wants to record and reuse it later.

A technique has been devised that enables generation of a first image even in a display device that displays data handwritten with a pen or a finger on a touch panel (see, for example, Patent Document 1). Patent Literature 1 discloses a display device that stores a first image of handwritten data.

However, the conventional first image has a problem that the user cannot edit the image of each application software displayed on the display or the like (hereinafter referred to as the second image) later. The display device provides functions such as displaying handwritten data by executing application software for handwriting (hereinafter simply referred to as the first application), but applications other than the first application (hereinafter referred to as the second application) are also provided. can run. Therefore, when the first image is generated while the second application is running, the first image including the second image generated by the second application is generated.

Since the second image partially hides the first image, the user may want to move or delete the second image. I couldn't.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a display device or the like capable of displaying a second image that can be edited by a user.

In view of the above problems, the present invention provides a display device for displaying a second screen generated by a second application superimposed on a first screen generated by a first application, wherein the first screen displayed on the first screen is displayed on the first screen. a whole image acquisition unit that acquires an image; a second image acquisition unit that acquires a second image displayed on the second screen; and a display processing unit that displays the second image superimposed on the first image. , an operation processing unit that receives edits to the second image, and a handwritten data processing unit that converts handwritten data with a pen or hand into a handwritten data image, and the first screen displays the handwritten data image and When an input image input from the outside is superimposed and displayed, and an operation for obtaining the first image is input, the input image obtaining unit obtains the input image, and the second image obtaining unit obtains the first image. obtaining two images, wherein the handwritten data processing unit arranges the second image on the handwritten data image, and the display processing unit overlaps the handwritten data image in which the second image is arranged and the input image; It is characterized by displaying

A display device or the like can be provided that can display a second image that can be edited by the user.

It is an example of the figure explaining the schematic operation|movement of the display apparatus which is a comparative example. It is a figure which shows an example of the 1st image which a display apparatus produces|generates. 1 is an example of an overall configuration diagram of an image processing system; FIG. 1 is an example of a hardware configuration diagram of a display device; FIG. It is an example of the functional block diagram which divides the function of a display apparatus into a block form, and demonstrates it. It is an example of the block diagram of each image layer. It is a figure which shows an example of a software configuration. FIG. 10 is an example of a functional block diagram showing functions of a display device related to saving a second image divided into blocks; It is a figure explaining arrangement|positioning of the 2nd image in a handwritten data image. FIG. 3 is a diagram illustrating an example of functions provided by an OS; FIG. It is a figure which shows an example of an application registration screen. It is an example of the flowchart figure which shows the procedure which a display apparatus produces|generates a 1st image. FIG. 13 is an example of a flowchart diagram showing a procedure of processing in step S2 of FIG. 12; It is a figure which shows an example of the handwritten data image in which two 2nd images are arranged. FIG. 13 is an example of a flowchart diagram showing the procedure of the processing in step S2 of FIG. 12 when the second application is not registered in advance in the registered application storage unit; FIG. 13 is a flow chart diagram showing a procedure of processing in step S2 of FIG. 12 using a second application registered in a registered application storage unit; FIG. 10 is a diagram showing another configuration example of the display device; FIG. 10 is a diagram showing another configuration example of the display device; FIG. 10 is a diagram showing another configuration example of the display device; FIG. 10 is a diagram showing another configuration example of the display device;

A display device and an image display method performed by the display device will be described below as an example of a mode for carrying out the present invention.

<Comparative example>
First, a suitable comparative example will be described when describing the display device of the present embodiment. Note that this comparative example is not necessarily prior art.

FIG. 1 is an example of a diagram explaining a schematic operation of a display device as a comparative example. FIG. 1(a) is an example of a screen displayed on the display 3 by the display device (hereinafter referred to as a first screen 9). The first screen 9 is displayed by a first application that displays data handwritten by the user on the touch panel with a pen or finger. Two second applications form windows superimposed on the first screen 9, and the applications display respective second screens 8A and 8B in the windows. As shown in FIG. 1(a), the second screens 8A and 8B overlap the handwritten data. A window is a rectangular area for displaying information provided on the display screen.

An arbitrary second image of the second images 11A and 11B is hereinafter referred to as a second image 11, and an arbitrary second screen of the second screens 8A and 8B is referred to as a second screen 8 hereinafter.

When the user inputs an operation to the display device to generate the first image 12 in which the first screen 9 is copied, the display device generates the first image 12 . FIG. 1B shows an example of the first image 12 as a comparative example. The first image 12 generated by the display device includes the second images 11A and 11B, but the display device is unaware of the existence of the second screens 8A and 8B. It is a part (integrated). Therefore, even if the user wants to check the handwritten data, editing such as moving or reducing the size of the second image 11 is not possible.

<Outline of operation of the display device of the present embodiment>
Next, schematic operation of the display device of this embodiment will be described with reference to FIG. FIG. 2 is a diagram for explaining the first image 12 generated by the display device of this embodiment. FIG. 2(a) is the same as FIG. 1(a).

When the user inputs an operation to generate the first image 12 of the first screen 9 to the display device, the display device generates the first image 12 and the second image 11 . FIG. 2B schematically shows the first image 12 and the second image 11. FIG. The display device detects the running second application and saves the second screens 8A and 8B generated by the second application as images different from the first image 12 . Second images 11A and 11B are generated separately from the first image 12, as shown in FIG. 2(b). The first image 12 and the second images 11A and 11B are separate images (for example, a total of three images are generated).

FIG. 2C shows the second images 11A and 11B superimposed on the first image 12 and displayed. Since the first image 12 and the second images 11A and 11B are independent images, the user can edit the second images 11A and 11B on the first image 12 by moving, reducing, enlarging, deleting, or rotating them. and the handwritten data under the second images 11A, 11B can be confirmed.

Thus, the display device of the present embodiment saves the second screen 8 as the second image 11 separately from the first image 12, so editing such as moving or reducing the size of the second image 11 is possible. become.

<Terms>
The first screen 9 is a screen generated by the first application. The first application is an application that provides the main functions of the display device, and mainly generates the first screen 9 covering almost the entire display 3 . The second screen 8 is a screen generated by an application other than the first application (this is called a second application). The second screen 8 overlaps the first screen 9 . However, the first screen 9 does not have to cover the entire display 3 .

Handwritten data is data in which a sequence of coordinate points is displayed as a trajectory on the display by the user continuously moving the input means. A series of operations in which the user presses the input means against the display, moves it continuously, and then releases it from the display is called a stroke, and data handwritten by the stroke is sometimes called stroke data. Handwritten data has one or more stroke data, but in this embodiment, it is called handwritten data without distinction.

<Overview of the system>
FIG. 3 is an overall configuration diagram of the image processing system 1 of this embodiment. In addition, in order to simplify the explanation, FIG. An electronic pen or the like may be used.

As shown in FIG. 3, the image processing system 1 includes a plurality of display devices 2a and 2b, a plurality of electronic pens 4a and 4b, USB memories 5a and 5b, external devices 6a and 6b, and a television (video) conference terminal. 7a, 7b, and a PC (Personal Computer) 8. Also, the display devices 2a, 2b and the PC 8 are connected via a communication network N so as to be able to communicate with each other. Furthermore, displays 3a and 3b are provided in the plurality of display devices 2a and 2b, respectively.

In addition, the display device 2a can display an image drawn by an event caused by the electronic pen 4a (touching the tip of the electronic pen 4a or the tip of the electronic pen 4a on the display 3a). can be done. It is also possible to change the image displayed on the display 3a based not only on the electronic pen 4a but also on the basis of events (gestures such as enlargement, reduction, page turning, etc.) generated by the user's hand Ha or the like.

A USB memory 5a can be connected to the display device 2a. The display device 2a can read electronic files such as PDF files from the USB memory 5a, and the display device 2a can record electronic files in the USB memory 5a. can be done. In addition, the display device 2a is provided with a cable 10a1 capable of communication according to standards such as DisplayPort (registered trademark), DVI (Digital Visual Interface), HDMI (registered trademark: High-Definition Multimedia Interface), and VGA (Video Graphics Array). An external device 6a is connected via. Cable 10a1 may be a network.

Then, the display device 2a generates an event by touching the display 3a, and transmits event information indicating the event to the external device 6a in the same manner as an event from an input device such as a mouse or a keyboard. Similarly, a television (video) conference terminal 7a is connected to the display device 2a via a cable 10a2 capable of communication according to the above standards. Note that the external device 6a and the video conference terminal 7a may communicate with the display device 2a by wireless communication conforming to various wireless communication protocols such as Bluetooth (registered trademark).

On the other hand, at other bases where the display device 2b is installed, similarly to the above, the display device 2b having the display 3b, the electronic pen 4b, the USB memory 5b, the external device 6b, the video conference terminal 7b, the cable 10b1, the cable 10b2 is utilized. Furthermore, it is also possible to change the image displayed on the display 3b based on an event caused by the user's hand Hb or the like.

As a result, an image drawn on the display 3a of the display device 2a at one site is also displayed on the display 3b of the display device 2b at another site, and vice versa. is displayed on the display 3a of the display device 2a at one site. As described above, the image processing system 1 can perform remote sharing processing for sharing the same image at a remote location, and is therefore very convenient when used for a conference or the like at a remote location.

In addition, hereinafter, when an arbitrary display device 2 among the plurality of display devices 2 is indicated, it is indicated as "display device 2". When indicating an arbitrary display 3 among the plurality of displays 3, it is indicated as "display 3". When indicating an arbitrary electronic pen among a plurality of electronic pens, it is indicated as "electronic pen 4". When indicating an arbitrary USB memory among a plurality of USB memories, it is indicated as "USB memory 5". When indicating an arbitrary external device among a plurality of external devices, it is indicated as "external device 6". When indicating an arbitrary teleconference terminal among a plurality of teleconference terminals, it is indicated as "teleconference terminal 7". Also, when indicating an arbitrary hand among a plurality of users' hands, it is indicated as "hand H". When indicating an arbitrary cable among a plurality of cables, it is indicated as "cable 10".

In addition, in the present embodiment, the display device 2 will be described as an example of an information processing device, but it is not limited to this. It may also be a telestrator used in a telescope, or a remote image (video) diagnostic device or the like. An example of the external device 6 is a notebook PC. Examples of the external device 6 include a desktop PC, a tablet PC, a smartphone, a PDA (Personal Digital Assistant), a digital video camera, a digital camera, a game machine, and the like. It may be a terminal capable of supplying frames. A predetermined application (program) for the display device 2 is running in these information processing devices, and the application displays handwritten data based on the touch position on the touch panel.

Furthermore, the communication network N includes the Internet, a LAN (Local Area Network), a mobile phone communication network, and the like. Further, in this embodiment, a USB memory is described as an example of a recording medium, but it is not limited to this. good.

<Hardware Configuration of Display Device>
FIG. 4 is an example of a hardware configuration diagram of a display device. In FIG. 4, the display device 2 is assumed to be an electronic blackboard. As shown in FIG. 4, the display device 2 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, an SSD (Solid State Drive) 204, a network I/ F205 and an external device connection I/F (Interface) 206 are provided.

Among these, the CPU 201 controls the operation of the entire display device 2 . The ROM 202 stores programs used to drive the CPU 201, such as the CPU 201 and an IPL (Initial Program Loader). A RAM 203 is used as a work area for the CPU 201 . The SSD 204 stores various data such as electronic blackboard programs. A network I/F 205 controls communication with the communication network N. FIG. The external device connection I/F 206 is an interface for connecting various external devices. The external devices in this case are, for example, a USB (Universal Serial Bus) memory 230 and external devices (microphone 240, speaker 250, camera 260).

The display device 2 also includes a capture device 211, a GPU 212, a display 3 controller 213, a contact sensor 214, a sensor controller 215, an electronic pen controller 216, a short-range communication circuit 219, an antenna 219a of the short-range communication circuit 219, and a power switch 222. , selection switches 223 , and a camera 224 .

Among these devices, the capture device 211 causes the display 3 of an external PC (Personal Computer) 270 to display video information as a still image or moving image. A GPU (Graphics Processing Unit) 212 is a semiconductor chip that specializes in graphics. A display 3 controller 213 controls and manages screen display in order to output an output image from the GPU 212 to the display 3 or the like. The contact sensor 214 detects that the electronic pen 290 , the user's hand H, or the like touches the display 3 . A sensor controller 215 controls the processing of the contact sensor 214 . The contact sensor 214 performs coordinate input and coordinate detection using an infrared shielding method. In this method of inputting coordinates and detecting coordinates, two light emitting/receiving devices installed at both ends of the upper side of the display 3 radiate a plurality of infrared rays parallel to the display 3, and are provided around the display 3. This is a method of receiving light that is reflected by a reflecting member and returns along the same optical path as that of light emitted by a light receiving element. The contact sensor 214 outputs to the sensor controller 215 the IDs of the infrared rays emitted by the two light emitting/receiving devices blocked by the object, and the sensor controller 215 identifies the coordinate position, which is the contact position of the object. The electronic pen controller 216 communicates with the electronic pen 290 to determine whether or not the display 3 has been touched with the tip of the pen or the bottom of the pen. The short-range communication circuit 219 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The power switch 222 is a switch for switching ON/OFF of the power of the display device 2 . The selection switches 223 are, for example, a group of switches for adjusting the brightness, color, etc. of the display on the display 3 .

A camera 224 is an imaging device built in, for example, a bezel of the display 3 and captures an image of the front of the display device 2 . If the camera 224 is present, an external camera like the camera 260 may be omitted.

Furthermore, the display device 2 has a bus line 210 . A bus line 210 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 201 shown in FIG.

In addition, the contact sensor 214 is not limited to the infrared blocking type, but is a capacitive type touch panel that identifies the contact position by detecting a change in capacitance, or identifies the contact position by a voltage change of two opposing resistive films. Various detection means may be used, such as a resistive touch panel that detects an electromagnetic induction generated when a contact object touches the display unit and identifies the contact position. Further, the electronic pen controller 216 may determine whether or not the part of the electronic pen 290 gripped by the user or other parts of the electronic pen is touched, in addition to the tip and the pen butt of the electronic pen 290 .

<Functional Configuration of Display Device>
Next, the functional configuration of the display device 2 will be described with reference to FIGS. 5 and 6. FIG. FIG. 5 is a functional block diagram for explaining the functions of the display device 2 divided into blocks, and FIG. 6 is an example of a configuration diagram of each image layer.

The display device 2 has each functional configuration shown in FIG. 5 by the hardware configuration and programs shown in FIG. The display device 2 can be a "sponsor device" that first starts the remote sharing process, and can also be a "participating device" that later participates in the already started remote sharing process. Further, the display device 2 is roughly divided into both a client section 20 and a server section 90 .

The client unit 20 and the server unit 90 are functions implemented within one housing of the display device 2 . When the display device 2 serves as the host device, the display device 2 implements the client section 20 and the server section 90 . Further, when the display device 2 is a participating device, the display device 2 implements the client section 20 but does not implement the server section 90 . That is, in FIG. 3, when the display device 2a is the host device and the display device 2b is the participating device, the client section 20 of the display device 2a can communicate with other devices via the server section 90 realized in the same display device 2a. communicates with the client unit 20 of the display device 2b. On the other hand, the client section 20 of the display device 2b communicates with the client section of the other display device 2a via the server section 90 implemented in the other display device 2a.

[Functional configuration of the client part]
Next, the functional configuration of the client unit 20 will be described mainly with reference to FIGS. 5 and 6. FIG. The client unit 20 includes an image acquisition unit 21, a coordinate detection unit 22, an automatic adjustment unit 23, a contact detection unit 24, an event distribution unit 25, an operation processing unit 26, a gesture processing unit 27, an image superimposition unit 28, and an image processing unit 30. , and a communication control unit 60 .

Of these, the video acquisition unit 21 acquires the output image of the external device 6 connected to the cable 10 . When the image acquisition unit 21 receives the image from the external device 6, it analyzes the image, and determines the resolution of the image frame, which is the display image of the external device 6 formed by the image, the update frequency of the image frame, and the like. Image information is derived and output to the image acquisition unit 31 .

The coordinate detection unit 22 detects the coordinate position where the user touches the input means on the display 3 . Handwritten data is the trajectory of the user's continuous movement of the input means on the display 3 . Also, this handwritten data may be called a stroke. Either the electronic pen 4 or the user's hand H may be used as the input means. The coordinate detection unit 22 also detects the touched area.

The automatic adjustment unit 23 is started when the display device 2 is started (it may be expressed as a restart), and the coordinates are detected by an optical sensor method so that the coordinate detection unit 22 can output an appropriate value. It adjusts the parameters when the detection unit 22 processes the image of the sensor camera.

The contact detection unit 24 detects an event caused by the user (such as an operation in which the tip of the electronic pen 4 or the tip of the electronic pen 4 is pressed (touched) on the display 3).

The event distribution unit 25 distributes the coordinate position of the event detected by the coordinate detection unit 22 and the detection result detected by the contact detection unit 24 to each event of handwritten data drawing, UI operation, and gesture operation. Here, "drawing handwritten data" means that the user presses the electronic pen 4 on the display 3 when the handwritten data image (B) shown in FIG. This is an event from moving the electronic pen 4 in this depressed state to finally releasing the electronic pen 4 from the display 3 . By drawing the handwritten data, for example, alphabets such as "S" and "T" are drawn on the display 3. FIG. Note that this "drawing of handwritten data" includes not only events for drawing an image, but also events for deleting an already drawn image and editing a drawn image.

"UI operation" is an event in which the user presses a predetermined position with the electronic pen 4 or hand H when a UI image (A) shown in FIG. 6, which will be described later, is displayed on the display 3. . By this UI operation, for example, the color, width, etc. of a line drawn with the electronic pen 4 are set.

"Gesture operation" is an event in which the user touches or moves the hand H on the display 3 when a handwritten data image (B) shown in FIG. 6 is displayed on the display 3. is. With this gesture operation, for example, by moving the hand H while the user touches the display 3, the image can be enlarged (or reduced), the display area can be changed, or the page can be switched. can.

The operation processing unit 26 executes various operations according to the UI element for which the event is generated, based on the UI operation determined by the event allocation unit 25 . Elements of this UI include, for example, buttons, lists, check boxes, and text boxes. The gesture processing unit 27 executes an operation corresponding to the gesture operation determined by the event distribution unit 25 .

The video superimposing unit 28 displays the image superimposed by the display processing unit 36, which will be described later, on the display unit 29 as a video. A display unit 29 indicates a display function realized by the display 3 . Further, the video superimposing unit 28 picture-in-pictures the video sent from the teleconference terminal 7 or the like to the video from the external device 6 or the like. Further, the image superimposing unit 28 performs switching to display the image that has been picture-in-picture and displayed on a part of the display unit 29 on the entire display unit 29 .

The image processing unit 30 performs superimposition processing and the like for each image layer as shown in FIG. The image processing unit 30 includes an image acquisition unit 31, a handwritten data processing unit 32, a menu screen generation unit 33, an image generation unit 34, a layout management unit 35, a display processing unit 36, a page processing unit 37, a file processing unit 40, a page It has a data storage unit 300 and a remote license management table 310 .

Of these, the image acquisition section 31 acquires each frame as an image from the video acquired by the video acquisition section 21 . The image acquisition section 31 outputs this image data to the page processing section 37 . This image corresponds to the input image (C) from the external device 6 or the like shown in FIG. In this embodiment, the input image (C) is called video.

The handwritten data processing unit 32 draws an image (handwritten data), deletes the drawn image, or deletes the drawn image, based on the handwritten data drawing event assigned by the event allocation unit 25. To edit. This handwritten data drawing image corresponds to the handwritten data image (B) shown in FIG. Further, each result of image drawing, deletion, and editing based on this handwritten data drawing is stored in the data management unit 80 as operation data, which will be described later.

The menu screen generation unit 33 generates a UI image (A) (user interface image) preset in the display device 2 . This UI image (A) corresponds to the UI image (A) shown in FIG.

The image generation unit 34 receives from the page processing unit 37 the media data included in the page data read by the page processing unit 37 from the page data storage unit 300 (see Tables 1 to 4). The image generator 34 outputs the received media data to the display processor 36 . An image based on this media data corresponds to the background image (D) shown in FIG. The pattern of the background image (D) is plain, grid display, or the like.

The layout management unit 35 provides the display processing unit 36 with layout information indicating the layout of each image output from the image acquisition unit 31, the handwritten data processing unit 32, and the menu screen generation unit 33 (or the image generation unit 34). is managing As a result, the layout management unit 35 instructs the display processing unit 36 to display the input image (C) and the handwritten data image (B) at positions in the UI image (A) and the background image (D). Or you can instruct whether to hide it.

Based on the layout information output from the layout management unit 35, the display processing unit 36 determines the layout of each image output from the image acquisition unit 31, the handwritten data processing unit 32, the menu screen generation unit 33, and the image generation unit 34. conduct.

The page processing unit 37 combines the data of the handwritten data image (B) and the data of the input image (C) into one page data and stores it in the page data storage unit 300 . The data of the handwritten data image (B) forms part of the page data as handwritten arrangement data (each handwritten data). The data of the input image (C) form part of the page data as media data. Then, when this media data is read from the page data storage unit 300, it is handled as data of the background image (D).

Further, the page processing unit 37 transmits the media data of the temporarily stored page data to the display processing unit 36 via the image generation unit 34, so that the image superimposition unit 28 displays the background image (D). 3 can be redisplayed. In addition, the page processing unit 37 returns the handwritten arrangement data (each handwritten data) of the page data to the handwritten data processing unit 32, so that the handwritten data can be re-edited. Furthermore, the page processing unit 37 can also delete or duplicate page data.

That is, the data of the input image (C) displayed on the display 3 when the page processing unit 37 stores the page data in the page data storage unit 300 is temporarily stored in the page data storage unit 300, and then When read from the page data storage unit 300, it is read as media data representing the background image (D). Then, page processing unit 37 outputs handwritten arrangement data representing the handwritten data image (B) among the page data read from page data storage unit 300 to handwritten data processing unit 32 . The page processing unit 37 also outputs media data indicating the background image (D) among the page data read from the page data storage unit 300 to the image generation unit 34 .

The display processing unit 36 receives an input image (C) from the image acquisition unit 31, a handwritten data image (B) from the handwritten data processing unit 32, a UI image (A) from the menu screen generation unit 33, and an image generation unit. A background image (D) from 34 is superimposed according to the layout specified by the layout management section 35 . As a result, as shown in FIG. 6, the UI image (A), the handwritten data image (B), the input image (C), and the background image (D) are displayed in the order that the user can see even if the images overlap. It consists of each layer.

Further, the display processing unit 36 switches between the input image (C) and the background image (D) shown in FIG. 6, and superimposes them exclusively on the UI image (A) and the handwritten image (B). is also possible. For example, when the UI image (A), the handwritten image (B), and the input image (C) are initially displayed and the cable 10 between the display device 2 and the external device 6 or the like is unplugged, By designation of the layout management unit 35, the input image (C) can be excluded from the superimposition target and the background image (D) can be displayed. In this case, the display processing unit 36 also performs display enlargement, display reduction, and display area movement processing.

The file processing unit 40 performs processing related to file input and output. The file processing unit 40 reads image data from a storage device such as the USB memory 5 and stores each page in the page data storage unit 300 as page data. Further, the file processing unit 40 converts the page data stored in the page data storage unit 300 into a PDF format file, and stores the file in a storage device such as the USB memory 5 or the like. In addition, a PDF file is attached to an e-mail and sent.

[Functional configuration of the server part]
The server section 90 is provided in each display device 2, and any display device 2 can serve as a server section. Therefore, the server section 90 has a communication control section 70 and a data management section 80 .

The communication control unit 70 controls communication with the communication control unit 70 in the client unit 20 in the same display device 2 and communication with the communication control unit 70 in the client unit 20 in another display device 2 via the communication network N. The data management unit 80 manages operation data, image data, and the like.

ページデータ記憶部３００は、表１に示されているようなページデータを記憶する。表１は、ページデータを概念的に示す。ページデータは、ディスプレイ３に表示される１ページ分のデータ（手書き配列データ（各手書きデータ）及びメディアデータ）である。なお、ページデータに含まれるパラメータの種類が多いため、ここでは、表１～表４に分けて、ページデータの内容を説明する。

The page data storage unit 300 stores page data as shown in Table 1. Table 1 conceptually shows page data. The page data is data for one page displayed on the display 3 (handwritten arrangement data (each handwritten data) and media data). Since there are many types of parameters included in the page data, the contents of the page data will be described separately in Tables 1 to 4 here.

The page data, as shown in Table 1, includes a page data ID for identifying an arbitrary page, a start time indicating the time when the display of this page was started, and rewriting of the content of the page by handwriting, gestures, or the like. end time indicating the time at which the is no longer performed, a handwritten array data ID for identifying handwritten array data generated by handwriting with the electronic pen 4 or the user's hand H, and a media data ID for identifying media data. are stored. The handwritten arrangement data is data for displaying the handwritten data image (B) shown in FIG. Media data is data for displaying the background image (D) on the display 3 .

With such page data, for example, when the user draws the alphabet "S" with the electronic pen 4, it is written in one stroke, so one handwritten data ID indicates one alphabet [S]. However, when the user draws the alphabet "T" with the electronic pen 4, it is written in two strokes, so two handwritten data IDs indicate one alphabet "T".

また、手書き配列データは、表２に示されているように詳細な情報を示している。表２は、手書き配列データを概念的に示す。表２に示されているように、１つの手書き配列データは、複数の手書きデータによって表される。そして、１つの手書きデータは、この手書きデータを識別するための手書きデータＩＤ、１つの手書きデータの書き始めの時刻を示す開始時刻、１つの手書きデータの書き終わりの時刻を示す終了時刻、手書きデータの色、手書きデータの幅、及び、手書きデータの通過点の配列を識別するための座標配列データＩＤを示している。

Also, the handwritten sequence data shows detailed information as shown in Table 2. Table 2 conceptually shows the handwritten sequence data. As shown in Table 2, one piece of handwritten array data is represented by a plurality of pieces of handwritten data. One piece of handwritten data includes a handwritten data ID for identifying this handwritten data, a start time indicating the time at which writing of the piece of handwritten data began, an end time indicating the time at which writing of the piece of handwritten data ended, and handwritten data. , the width of the handwritten data, and the coordinate array data ID for identifying the array of passing points of the handwritten data.

更に、この座標配列データは、表３に示されているように詳細な情報を示している。表３は、座標配列データを概念的に示す。表３に示されているように、座標配列データは、ディスプレイ３上の１点（Ｘ座標値、Ｙ座標値）、この１点を通過したときの手書きの開始時刻からの差分の時刻（ｍｓ）、及び、この１点における電子ペン４の筆圧の各情報を示している。即ち、表３に示されている１点の集まりが、表２に示されている１つの座標配列データで示されている。例えば、ユーザが電子ペン４によってアルファベット「Ｓ」を描く場合、一筆書きとなるが、「Ｓ」を描き終えるまでに、複数の通過点を通過するため、座標配列データは、これら複数の通過点の情報を示している。

Furthermore, this coordinate array data shows detailed information as shown in Table 3. Table 3 conceptually shows the coordinate array data. As shown in Table 3, the coordinate array data consists of one point (X coordinate value, Y coordinate value) on the display 3 and the difference time (ms ), and the writing pressure of the electronic pen 4 at this one point. That is, a set of points shown in Table 3 is represented by one coordinate array data shown in Table 2. For example, when the user draws the alphabet "S" with the electronic pen 4, it is written in one stroke. information.

また、表１に示されているページデータのうちのメディアデータは、表４に示されているように詳細な情報を示している。表４は、メディアデータを概念的に示す。表４に示されているように、メディアデータは、表１に示されているページデータにおけるメディアデータＩＤ、メディアデータのデータ種類、ページ処理部３７からページデータ記憶部３００にページデータが記憶された記録時刻、ページデータによってディスプレイ３上に表示される画像の位置（Ｘ座標値、Ｙ座標値）及び画像のサイズ（幅、高さ）、並びにメディアデータの内容を示すデータが関連付けられて示されている。このうち、ページデータによってディスプレイ３上に表示される画像の位置は、ディスプレイ３の左上端の座標を（Ｘ座標値，Ｙ座標値）＝（０，０）とした場合に、ページデータによって表示される画像の左上端の位置を示している。

Also, the media data of the page data shown in Table 1 indicates detailed information as shown in Table 4. Table 4 conceptually shows media data. As shown in Table 4, the media data includes the media data ID in the page data shown in Table 1, the data type of the media data, and the page data stored in the page data storage unit 300 from the page processing unit 37. The recorded time, the position (X coordinate value, Y coordinate value) and size (width, height) of the image displayed on the display 3 by the page data, and the data indicating the contents of the media data are displayed in association with each other. It is Among them, the position of the image displayed on the display 3 by the page data is displayed by the page data when the coordinates of the upper left corner of the display 3 are (X coordinate value, Y coordinate value)=(0, 0). It indicates the position of the upper left corner of the image to be displayed.

<Functions of the display device related to saving the second image of the second application>
Next, with reference to FIGS. 7 and 8, functions of the display device 2 regarding saving of images of the second application will be described.

FIG. 7 is a diagram showing an example of a software configuration. One first application 102 and one or more second applications 103 are installed in the display device 2 . As shown in FIG. 7, first application 102 and second application 103 operate on OS 101 . The OS 101 is basic software that manages the display device 2 and provides an API (application interface) for the first application 102 or the second application 103 to use the hardware or software of the display device 2 . The OS 101 may be, for example, Windows (registered trademark), UNIX (registered trademark), LINUX (registered trademark), MAC OS (registered trademark), Android (registered trademark), iOS (registered trademark), or the like.

The first application 102 is an application that provides the main functions of the display device 2 as described above, and performs display and storage of handwritten data.

The second application 103 may be any application that may be used together with the first application 102 . In this embodiment, since the second image 11 is saved, it is assumed that the application is not an application that operates only in the background, but an application that displays some kind of UI (second screen 8). For example, a videophone application, a file viewer application, a schedule viewer application, or a browser application. At least one second application 103 is sufficient.

Moreover, it is more preferable that the second application 103 is a cooperative application that cooperates with the first application 102 . A cooperative application is an application that is frequently used by the user when using the first application 102, and is registered in advance by the user. When the first application 102 is activated, the cooperative application can also be activated.

Software (more precisely, computer software) refers to programs related to computer operations and other information used for processing by a computer, which conforms to programs. Application software is a general term for software used to perform a specific task among software classifications. On the other hand, an operating system (OS) is software that controls a computer and enables application software and the like to use computer resources. It is often abbreviated as "OS".

The operating system performs basic management and control of the computer, such as control of input/output, management of hardware such as memory and hard disk, and management of processes. Application software operates using functions provided by the operating system.

A program is a set of instructions for a computer that are combined to produce a result. Also, what conforms to a program refers to something that cannot be called a program because it is not a direct command to a computer, but has properties similar to a program in that it defines the processing of the computer. For example, a data structure (a logical structure of data represented by interrelationships between data elements) corresponds to a program.

Each of the above programs may be recorded in a computer-readable recording medium in an installable or executable format and distributed. Examples of recording media include CD-Rs (Compact Disc Recordable), DVDs (Digital Versatile Disks), Blu-ray discs, SD cards, and the like. Also, the recording medium can be provided domestically or internationally as a program product.

FIG. 8 is an example of a functional block diagram showing functions of the display device 2 related to saving the second image 11 divided into blocks. The display device 2 in FIG. 8 extracts and shows some functions related to saving the second image 11 from the functions shown in FIG.

The operation processing unit 26 receives a save operation of the first screen 9 performed by the user on the UI image (A). In this embodiment, the save operation on the first screen 9 is assumed to include the save operation on the second screen 8 as well. Only the first screen 9 may be saved, or only the second screen 8 may be saved. The details of the operation are notified to the image acquisition unit 31 , the image generation unit 34 and the handwritten data processing unit 32 . The operation processing unit 26 also accepts editing of the second image 11 displayed on the display 3 .

A handwritten data image (B) and a UI image (A) are displayed on the display 3 while an input image (C) or a background image (D) is displayed. The objects to be saved are the handwritten data image (B), the input image (C) and the background image (D). However, since the input image (C) and the background image (D) are not displayed at the same time, the acquired image is
・Input image (C) only ・Background image (D) only ・Handwritten data image (B) and input image (C)
・Handwritten data image (B) and background image (D)
is. Also, the background image (D) may not be acquired as a function of the display device 2 .

All of these are images generated by the first application 102 and correspond to the first image 12 . In the present embodiment, it is further assumed that the second screen 8 generated by the second application 103 is superimposed on the first screen 9 and displayed.

The whole image acquisition unit 51 acquires the first image 12 displayed on the first screen 9 and the second image 11 displayed on the second screen 8 . As described above, the first screen 9 includes the handwritten data image (B) and the input image (C) or background image (D). Conventionally, the handwritten data image (B) and the input image (C) or the background image (D) were saved in image data such as PDF data, but in this embodiment the input image (C) or the background image (D) is Although it is saved as image data such as PDF, handwritten data may be saved as coordinate data.

The image acquisition section 31 has an input image acquisition section 52 and a second image acquisition section 53 . The input image obtaining unit 52 saves the input image (C) displayed on the display 3 (first screen 9) when the operation processing unit 26 accepts the saving operation. However, if the input image (C) is not input, it is not saved.

The second image acquisition unit 53 saves the second image 11 of the second screen 8 generated by the second application 103 currently displayed on the display 3 . The second application 103 is registered in the registered application storage unit 54 in advance. Since the second application 103 displays the second screen 8 in a rectangular area called a window, the image within this rectangular area is the second image 11 . The details of saving the second image 11 will be described later.

表５は、登録アプリ記憶部５４に記憶されている第二アプリ１０３を模式的に示す。登録アプリ記憶部５４にはアプリＩＤに対応づけて第二アプリ１０３のアプリ名が登録されている。このような第二アプリ１０３のアプリ名は、第二画像取得部５３が第二画像１１を保存する際に使用される。また、登録アプリ記憶部５４への第二アプリ１０３の登録は予めユーザが行う。

Table 5 schematically shows second applications 103 stored in registered application storage unit 54 . The application name of the second application 103 is registered in the registered application storage unit 54 in association with the application ID. Such an application name of the second application 103 is used when the second image acquisition unit 53 saves the second image 11 . Also, the user registers the second application 103 in the registered application storage unit 54 in advance.

Returning to FIG. 8 , the image generation section 34 has a background image acquisition section 56 . The background image obtaining unit 56 saves the background image (D) displayed on the display 3 (first screen 9) when the operation processing unit 26 accepts the save operation. However, when the input image (C) is input, the background image (D) is not saved (because the background image (D) is not in contact with the user's eyes).

Since the handwritten data processing unit 32 records the handwritten data in the page data storage unit 300 in real time as described above, the handwritten data need not be saved separately when the operation processing unit 26 receives the save operation. . The handwritten data processing unit 32 acquires one page of handwritten data via the page processing unit 37 when the operation processing unit 26 receives a save operation. This is for displaying the handwritten data image (B) together with the second image 11 .

The page processing unit 37 acquires the background image (D) acquired by the background image acquiring unit 56 or the input image acquired by the input image acquiring unit 52, and the second image 11 acquired by the second image acquiring unit 53. Each page is associated and stored in the image data storage unit 55 . Table 6 shows an example of the image data stored in the image data storage unit 55.

Further, the page processing unit 37 sends the input image (C) or the background image (D) stored in the image data storage unit 55 to the image generation unit 34, and transmits the handwritten data and the second image 11 to the handwritten data processing unit 32. send to The image generation unit 34 sends the input image (C) or the background image (D) to the display processing unit 36, and the handwritten data processing unit 32 converts the handwritten data into the handwritten data image (B) and arranges the second image 11. and sent to the display processing unit 36. Thereby, the display processing unit 36 can display the input image (C) or the background image (D), the second image 11, and the handwritten data image (B) in an overlapping manner.

表６は、画像データ記憶部５５に記憶されている画像データを模式的に示す。画像データ記憶部５５は、ページデータＩＤ、背景画像（Ｄ）、入力画像（Ｃ）、及び、第二画像１～ｎ、の各項目を有している。入力画像（Ｃ）と背景画像（Ｄ）はどちらかが記憶されていればよく、上記のメディアデータに相当する。表６では画像データそのものが画像データ記憶部５５に保存されているが、メディアデータへのパスで代用してよい。

Table 6 schematically shows the image data stored in the image data storage unit 55. The image data storage unit 55 has items of page data ID, background image (D), input image (C), and second images 1 to n. Either the input image (C) or the background image (D) may be stored, and corresponds to the above media data. Although the image data itself is stored in the image data storage unit 55 in Table 6, a path to the media data may be substituted.

The second images 1 to n are the second images 11 of the second screen 8 displayed in the window by the second application 103 . The second image 11 is also not saved when the second application 103 is not running.

<About the display of the saved second image>
The display device 2 displays the four layers (UI image (A), handwritten data image (B), input image (C), background image (D)) shown in FIG. The image 11 is placed on one of these layers. This saves memory without the need for more layers. In this embodiment, as an example, the display device 2 arranges the second image 11 in the handwritten data image (B). However, it may be placed in the UI image (A), the input image (C), or the background image (D).

FIG. 9 is a diagram illustrating the arrangement of the second image 11 in the handwritten data image. 9A shows the case where the second image 11 is not displayed, and FIG. 9B shows the case where the second image 11 is displayed.

As shown in FIG. 9A, for example, characters "AB" are handwritten in the handwritten data image (B). In this case, the handwritten data image (B) (handwritten layer) is drawn with the handwritten data 110 forming the characters "AB".

On the other hand, as shown in FIG. 9B, when the second image 11 of the second screen 8 that has been displayed overlapping the first screen 9 is saved, the handwritten data processing unit 32 processes the handwritten data image ( B) Place the second images 11A and 11B on the (handwritten layer).

In this way, coordinate data is arranged in the handwritten data image (B), and the second image 11 is arranged as a display component. Even if the user adds or deletes coordinate data later, it can be reflected in the handwritten data image (B). Since the display component is also not integrated with the handwritten data image (B), it can be edited separately from the handwritten data image (B). Therefore, the user can edit the second image 11 for the handwritten data image (B) in which the second image 11 is arranged.

<How to capture the second image>
In order for the second image acquisition unit 53 to acquire the second image 11, information regarding the second application 103 such as the position and size of the second image 11 is required. The second image acquisition unit 53 acquires information about the second application 103 using functions and commands provided by the OS 101 . In this embodiment, as an example, the case where the OS 101 is Windows (registered trademark) will be described.

FIG. 10 shows an example of functions provided by the OS 101. FIG. FIG. 10(a) shows the argument and return value of the GetProcessesByName() function. With the GetProcessesByName( ) function, the second image acquisition unit 53 can acquire the process ID of the application, which will be described later, from the application name (file name in FIG. 10A). The second image acquisition unit 53 sequentially reads the applications stored in the registered application storage unit 54, executes the GetProcessesByName() function, and acquires the process ID of each application.

A process is a unit of processing executed by a CPU when a program or the like is run on a computer.

When an application starts, the OS 101 issues a process ID for identifying this application. A process ID is a unique identifier for each process for identifying the process in which the OS 101 is currently running. It is used in Windows (registered trademark) and UNIX-based OS (registered trademark), and unsigned integer values are often used. If a program runs multiple processes, a value is assigned to each process. Also, when the same program is started multiple times, different values are assigned to distinguish between them.

In Windows (registered trademark), the own process ID is obtained by the function GetCurrentProcessId(), and the process IDs of other processes are obtained by the function GetProcessId(). Internally, the process ID is called the client ID and is assigned from the same namespace as the thread ID, so the two never overlap. A process ID of 0 is assigned to the so-called system idle process, and a process ID of 4 is assigned to the system process.

In Unix-based OS (registered trademark), the own process ID is obtained by the getpid() system call, and in the shell it is obtained by the variable $$. The process ID of the parent process can be obtained with the getpid() system call.

FIG. 10(b) shows the argument and return value of the GetWindowThreadProcessId() function. The GetWindowThreadProcessId( ) function is a function that obtains the process ID that generated this window from the window handle. A window handle is an identifier assigned to each window in Windows (registered trademark). This window handle is specified when an application uses the functions of the OS 101 to control the window.

The window handle is unknown when the second image acquisition unit 53 executes the GetWindowThreadProcessId( ) function. Therefore, the second image acquisition unit 53 acquires window handles of all applications executed by the display device 2 . First, the GetTopWindow() function obtains the topmost child window handle counting from the top level (desktop). From this window, use the GetNextWindow() function to trace the next window handle. This will get all the window handles.

When the GetWindowThreadProcessId() function is executed specifying the window handle obtained in this way, the second image obtaining unit 53 can obtain the process ID of the application that generated the window (the window displayed on the display 3). When the process ID acquired by the GetWindowThreadProcessId() function matches the process ID acquired by the GetProcessesByName() function, the second image acquisition unit 53 determines that the window handle of the second application 103 has been specified (GetWindowThreadProcessId() It turns out that the window handle specified by the function was the window handle of the second application 103).

FIG. 10(c) shows the argument and return value of the GetWindowInfo() function. The GetWindowInfo() function is a function that gets information about a window from a window handle. Execution of the GetWindowInfo() function gives a return value of WINDOWINFO. The client's rectangular area (the coordinates of the diagonal vertices of the rectangular area) is obtained in the rcClient member of WINDOWINFO. The client is the area excluding the frame of the window (part containing only the second image).

As described above, since the display position and display size of the second image 11 within the window can be identified, the second image acquisition unit 53 stores the second image 11 in the video memory holding the data to be displayed on the display 3. By specifying the position and size, the second image 11 generated by each application can be acquired from the video memory.

表７は、第二画像取得部５３が取得するアプリに関する情報を模式的に示す。アプリに関する情報は、プロセスＩＤ、アプリ名、ウィンドウハンドル、表示位置、及び、表示サイズを有している。プロセスＩＤはGetProcessesByName()関数により取得され、アプリ名は登録アプリ記憶部５４に記憶されているアプリのファイル名であり、ウィンドウハンドルはGetWindowThreadProcessId()関数により特定され、表示位置と表示サイズはGetWindowInfo（）関数により取得されている。なお、表示位置と表示サイズは矩形領域の対角頂点の座標でもよい。

Table 7 schematically shows information about apps acquired by the second image acquisition unit 53 . Information about the application has a process ID, application name, window handle, display position, and display size. The process ID is obtained by the GetProcessesByName() function, the application name is the file name of the application stored in the registered application storage unit 54, the window handle is specified by the GetWindowThreadProcessId() function, and the display position and display size are obtained by the GetWindowInfo() function. ) function. Note that the display position and display size may be the coordinates of the diagonal vertices of the rectangular area.

<Method of registering an application to be registered in the registered application memory>
FIG. 11 shows an example of the application registration screen 130. As shown in FIG. The application registration screen 130 is a screen for the user to register the second application 103 that cooperates with the first application 102 in the registered application storage unit 54 .

The application registration screen 130 may be displayed by the display device 2 or may be displayed by a terminal device such as a PC that communicates with the display device 2 via a network. A web browser is running on the terminal device, and the display device 2 serves as a server and transmits screen information of the application registration screen 130 to the terminal device.

The application registration screen 130 has a “display execution file list” button 131 , a list display field 132 , and a registration button 133 . When the user presses the “display execution file list” button 131 , the display device 2 searches the folder of the display device 2 for execution files (files with extension exe) and displays them in the list display field 132 . In this case, it is preferable that the folder in which the cooperative application is installed is determined. The display device 2 can search only the cooperative application by searching only the predetermined folder. The user selects the file name of the application in list display field 132 and presses registration button 133 . Thereby, the application is registered in the registered application storage unit 54 .

The application registration screen 130 also has an application name input field 134 , a search button 135 , a search result display field 136 and a registration button 137 . When the user knows the name of the application to be registered, the user enters part or all of the application name in the application name input field 134 and presses the search button 135 . The display device 2 searches the folder by the application name and displays the search result in the search result display field 136. FIG. When the desired application is displayed in search result display field 136 , the user selects the file name of the application and presses registration button 137 . Thereby, the application is registered in the registered application storage unit 54 .

<Operation procedure>
FIG. 12 is an example of a flow chart diagram showing a procedure for the display device 2 to generate the first image 12 . The process of FIG. 12 starts when the user performs a save operation on the first screen 9 for the UI image (A).

The input image acquisition unit 52 acquires the input image (C) and stores it in the image data storage unit 55 (S1). The background image acquisition unit 56 may store the background image (D). Also, the second image acquisition unit 53 acquires the second image 11 of each application and stores it in the image data storage unit 55 (S2). Details of the processing in step S2 will be described with reference to FIG. As a result, the input image (C) and the second image 11 of each second application 103 are stored in the image data storage unit 55 . Note that the processing order of steps S1 and S2 may be reversed.

Next, the page processing section 37 sends the input image (C) to the image generating section 34 and sends the handwritten data and the second image 11 to the handwritten data processing section 32 . The handwritten data processing unit 32 converts the handwritten data into the handwritten data image (B), and converts each second image 11 into the handwritten data image (B) based on the position (Table 7) obtained when the second image 11 is generated. to be placed. That is, the second image 11 is arranged at the same position as the second screen.

The handwritten data processing unit 32 sends the handwritten data image (B) in which the second image 11 is arranged to the display processing unit 36 , and the image generation unit 34 sends the input image (C) to the display processing unit 36 . As a result, the display processing unit 36 superimposes the input image (C) and the handwritten data image (B) in which the second image 11 is arranged and displays them on the display 3 (S3).

FIG. 13 is a flow chart showing the procedure of the processing in step S2 of FIG. The second image acquisition unit 53 reads the application name registered in the registered application storage unit 54 (S11).

Next, the second image acquisition unit 53 acquires the display position and display size of the second application 103 using various functions described with reference to FIG. 10 (S12).

The second image acquisition unit 53 stores the second image 11 of the second application 103 whose display position and display size are known in the image data storage unit 55 (S13). Instead of storing the second image 11 individually, the first image 12 of the entire first screen 9 (desktop screen) is generated, and the second image 11 is cropped based on the display position and display size of the second application 103. can be saved.

The second image acquisition unit 53 performs the process of FIG. 13 for all the second applications 103 registered in the registered application storage unit 54. FIG. The handwritten data processing unit 32 arranges all the second images 11 in the handwritten data image (B).

<Example of editing the second image>
FIG. 14 is an example of a handwritten data image (B) in which two second images 11 are arranged. FIG. 14A shows the handwritten data image (B) immediately after the first image 12 (for example, the input image (C) and the handwritten data image) is generated. A second image 11 is displayed at the same position.

The user deleted the second image 11A and reduced the second image 11B while keeping the first image 12 fixed. The first image 12 in this state is shown in FIG. 14(b). In this embodiment, the user can delete or reduce the size of the second image 11 in this way.

Further, the user moves the second images 11A and 11B from the state of FIG. 14(a) while the first image 12 remains fixed. The first image 12 in this state is shown in FIG. 14(c). In this embodiment, the user can move the second image 11 in this way.

Further, the user enlarges the second image 11A from the state of FIG. 14(c) while keeping the first image 12 fixed. The first image 12 in this state is shown in FIG. 14(d). In this embodiment, the user can thus enlarge the second image 11 .

<Another example of acquiring the display position and display size of the second application>
Although the second application 103 is registered in advance in the registered application storage unit 54 in this embodiment, the second application 103 does not have to be registered. In this case, the second image 11 can be generated by the second image acquiring unit 53 capturing all of the second screen 8 generated by the second application 103 that displays windows other than the first application 102 .

In this case, the second image acquisition unit 53 acquires the active window handle by the GetTopWindow( ) function and the GetNextWindow( ) function, for example. Since the window handle has been obtained, the second image obtaining unit 53 can obtain the display position and display size of the second application 103 by executing the GetWindowInfo( ) function. If there is a second application 103 whose display size is not zero, the second image acquisition unit 53 generates the second image 11 .

FIG. 15 is a flowchart showing the procedure of the processing in step S2 of FIG. 12 when the second application 103 is not registered in the registered application storage unit 54 in advance.

The second image acquisition unit 53 acquires the window handles of all running applications (S21). Next, the display position and display size of the second application 103 are acquired (S22).

Second image acquisition unit 53 determines whether the display size of second application 103 is not zero (S23).

When the display size of the second application 103 is not zero, the second image acquisition unit 53 acquires the second image 11 of the second application 103 for which the display position and display size of the second application 103 are known, and stores the second image 11 in the image data storage unit. 55 (S24).

The second image acquisition unit 53 performs the process of FIG. 15 for all the second applications 103 that are running. The handwritten data processing unit 32 arranges all the second images 11 in the handwritten data image (B).

<< Narrowing down the applications stored in the registered application memory >>
Alternatively, applications can be narrowed down by utilizing the fact that the second application 103 is registered in the registered application storage unit 54 .

FIG. 16 is a flow chart showing the procedure of the process of step S2 in FIG. 12 using the second application 103 registered in the registered application storage unit 54. As shown in FIG.

First, the second image acquisition unit 53 acquires the names of all running applications (S31). For example, in the case of windows (registered trademark), executing the command tasklist obtains a list of executable files (.exe) of applications that are currently running.

The second image acquisition unit 53 determines whether or not the execution file found in the tasklist is registered in the registered application storage unit 54 (S32).

When the application registered in the registered application storage unit 54 is found, the second image acquisition unit 53 acquires the display position and display size of the second application 103 (S33).

The second image acquisition unit 53 acquires the second image 11 of the second application 103 for which the display position and display size of the second application 103 are known, and stores the second image 11 in the image data storage unit 55 (S34).

The second image acquisition unit 53 performs the process of FIG. 16 for all the second applications 103 that are running. The handwritten data processing unit 32 arranges all the second images 11 in the handwritten data image (B).

In this manner, the second image acquisition unit 53 can identify the running second application 103 by various methods and generate the second image 11 .

<Another way to get the app name>
You may also use EnumWindows(EnumWindowCallback, 0). The first application 102 calls EnumWindowCallback for each existing top-level window handle by calling EnumWindows(EnumWindowCallback, 0). As a result, EnumWindowCallback(HWND handle, LPARAM lparam) is called for the number of existing window handles, so the first application 102 uses GetWindowThreadProcessId( ) in the Callback function to obtain the process ID for each handle. Furthermore, the process name for each process ID can also be obtained using GetModuleBasename() for each process ID obtained within this Callback function. By using EnumWinodws in this way, it is possible to obtain the process ID and process name for each window handle and determine whether the application names match.

<Summary>
As described above, the display device of the present embodiment saves the second screen 8 as the second image 11 separately from the first image 12, so the first image 12 and the second images 11A and 11B are independent images. can be In addition, since the second image 11 is arranged in the handwritten data image (B), there is no need to increase the number of layers, and editing such as moving or reducing the second image 11 is possible in the same manner as the handwritten data.

In this embodiment, a display device that displays an image by projecting it will be described.

<<Another example 1 of the configuration of the display device>>
Although the display device of this embodiment is described as having a large touch panel, the display device is not limited to having a touch panel.

FIG. 17 is a diagram showing another configuration example of the display device. In FIG. 17, a projector 411 is installed on the upper side of a normal whiteboard 413 . This projector 411 corresponds to a display device. The normal whiteboard 413 is not a flat panel display integrated with a touch panel, but a whiteboard on which the user directly writes with a marker. Note that the whiteboard may be a blackboard, and may be a flat surface that is large enough to project an image.

The projector 411 has an ultra-short focal length optical system, and can project an image with little distortion on the whiteboard 413 from about 10 cm. This image may be transmitted from the PC 400 - 1 connected wirelessly or by wire, or may be stored in the projector 411 .

The user writes by hand on the whiteboard 413 using the dedicated electronic pen 2501 . The electronic pen 2501 has a light-emitting portion, for example, at its tip, which is turned on when the user presses it against the whiteboard 413 for handwriting. Since the wavelength of light is in the near infrared or infrared, it is invisible to the user's eyes. The projector 411 has a camera, captures an image of the light emitting unit, analyzes the image, and identifies the direction of the electronic pen 2501 . Also, the electronic pen 2501 emits light and emits sound waves, and the projector 411 calculates the distance based on the arrival time of the sound waves. The position of the electronic pen 2501 can be identified by the direction and distance. Handwritten data is drawn (projected) at the position of the electronic pen 2501 .

Since the projector 411 projects the menu 430, when the user presses a button with the electronic pen 2501, the projector 411 identifies the pressed button based on the position of the electronic pen 2501 and the ON signal of the switch. For example, when the save button 431 is pressed, the data (set of coordinates) handwritten by the user is saved in the projector 411 . The projector 411 stores the handwritten information in a predetermined server 412, USB memory 5, or the like. Handwritten information is stored page by page. Since the coordinates are saved as they are instead of the image data, the user can re-edit them. However, in this embodiment, the menu 430 does not have to be displayed because the operation command can be called by handwriting.

<<Another example 2 of the configuration of the display device>>
FIG. 18 is a diagram showing another configuration example of the display device. In the example of FIG. 18, a terminal device 600, an image projection device 700A, and a pen motion detection device 810 are provided as display devices.

The terminal device 600 is connected to the image projection device 700A and the pen motion detection device 810 by wire. The image projection device 700A projects image data input from the terminal device 600 onto the screen 800 .

The pen motion detection device 810 communicates with the electronic pen 820 and detects motion of the electronic pen 820 in the vicinity of the screen 800 . Specifically, the electronic pen 820 detects coordinate information indicating a point indicated by the electronic pen 820 on the screen 800 and transmits the coordinate information to the terminal device 600 .

Terminal device 600 generates image data of handwritten data image (B) input by electronic pen 820 based on the coordinate information received from pen motion detection device 810, and projects handwritten data image (B) by image projection device 700A. Draw on the screen 800 .

In addition, the terminal device 600 generates superimposed image data indicating a superimposed image obtained by synthesizing the background image (D) projected by the image projection device 700A and the handwritten data image (B) input by the electronic pen 820. .

<<Another example 3 of the configuration of the display device>>
FIG. 19 is a diagram illustrating a configuration example of a display device. In the example of FIG. 19, a terminal device 600, a display 800A, and a pen motion detection device 810 are provided as display devices.

Pen motion detection device 810 is arranged near display 800 A, detects coordinate information indicating a point indicated by electronic pen 820 A on display 800 A, and transmits the information to terminal device 600 . Note that in the example of FIG. 19, the electronic pen 820A may be charged by the terminal device 600 via a USB connector.

Terminal device 600 generates image data of handwritten data image (B) input by electronic pen 820A based on the coordinate information received from pen motion detection device 810, and displays it on display 800A.

<<Another example 4 of the configuration of the display device>>
FIG. 20 is a diagram illustrating a configuration example of a display device. In the example of FIG. 20, a terminal device 600 and an image projection device 700A are provided as display devices.

Terminal device 600 performs wireless communication (Bluetooth (registered trademark), etc.) with electronic pen 820B to receive coordinate information of a point indicated by electronic pen 820B on screen 800 . Based on the received coordinate information, terminal device 600 generates image data of handwritten data image (B) input by electronic pen 820B, and causes image projection device 700A to project handwritten data image (B).

In addition, the terminal device 600 generates superimposed image data indicating a superimposed image obtained by synthesizing the background image (D) projected by the image projection device 700A and the handwritten data image (B) input by the electronic pen 820. .

As described above, each of the above-described embodiments can be applied in various system configurations.

<Other application examples>
Although the best mode for carrying out the present invention has been described above using examples, the present invention is by no means limited to such examples, and various modifications can be made without departing from the scope of the present invention. and substitutions can be added.

For example, in the present embodiment, a display device that can be used as an electronic blackboard has been described, but the display device may be any display device that can display images, such as digital signage.

Moreover, the image to be displayed may be a moving image or a still image.

Further, although the electronic blackboard has been described as an example in the present embodiment, any information processing apparatus having a touch panel can be suitably applied. Examples of information processing devices equipped with a touch panel include PJs (projectors), output devices such as digital signage, HUD (Head Up Display) devices, industrial machines, imaging devices, sound collectors, medical equipment, network appliances, A notebook PC (Personal Computer), a mobile phone, a smart phone, a tablet terminal, a game machine, a PDA (Personal Digital Assistant), a digital camera, a wearable PC, a desktop PC, or the like may be used.

Further, in the present embodiment, the coordinates of the pen tip are detected by the method of detecting the coordinates of the pen tip with the touch panel, but the coordinates of the pen tip may be detected by ultrasonic waves. The pen emits light and emits ultrasonic waves, and the display device 2 calculates the distance from the arrival time of the ultrasonic waves. The position of the pen can be specified by direction and distance. The projector draws (projects) the trajectory of the pen as handwritten data.

6 and 8 are divided according to main functions in order to facilitate understanding of the processing by the display device 2. FIG. The present invention is not limited by the division method or name of the processing unit. The processing of the display device 2 can also be divided into more processing units according to the content of the processing. Also, one processing unit can be divided to include more processing.

Also, each function of the embodiments described above can be realized by one or more processing circuits. Here, the "processing circuit" in this specification means a processor programmed by software to perform each function, such as a processor implemented by an electronic circuit, or a processor designed to perform each function described above. devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays) and conventional circuit modules.

1 image processing system 2 display device 3 display

Japanese Patent Application Laid-Open No. 2016-15098

Claims (8)

A display device for displaying a second screen generated by a second application superimposed on a first screen generated by a first application,
a whole image acquisition unit that acquires the first image displayed on the first screen;
a second image acquisition unit that acquires a second image displayed on the second screen;
a display processing unit that displays the second image superimposed on the first image;
an operation processing unit that receives edits to the second image;
a handwritten data processing unit that converts handwritten data with a pen or hand into a handwritten data image;
The handwritten data image and the input image input from the outside are superimposed and displayed on the first screen,
When an operation to acquire the first image is input,
an input image acquiring unit acquires the input image, the second image acquiring unit acquires the second image,
The handwritten data processing unit arranges the second image in the handwritten data image,
The display device, wherein the display processing unit displays the handwritten data image in which the second image is arranged and the input image in an overlapping manner . The second image acquisition unit acquires the position of the second screen in the first screen,
2. The display device according to claim 1, wherein the display processing unit displays the second image on the same position of the second screen on the first screen. 3. The display device according to claim 1, wherein the operation processing unit accepts reduction, enlargement, deletion, or movement of the second image while fixing the first image. The second image acquisition unit acquires a process ID of an application name of the second application registered in advance from the OS,
acquiring from the OS the display position of the second screen generated by the second application based on the window handle of the second application having a process ID that matches the process ID of the application running on the display device;
4. The display device according to any one of claims 1 to 3, wherein the second image is acquired from the acquired display position. The second image acquisition unit acquires window handles of all applications running on the display device from the OS,
obtaining from the OS the display position and display size of the window having the window handle;
identifying that the app with the window handle whose display size is not zero is the second app;
4. The display device according to any one of claims 1 to 3, wherein the second image is acquired from the display position of the second application acquired from an OS. The second image acquisition unit acquires application names of all applications running on the display device from the OS,
When the application name obtained from the OS matches the application name of the pre-registered second application,
Acquiring the display position of the application with the application name from the OS,
4. The display device according to claim 1, wherein said second image is obtained from said display position. An image display method performed by a display device that displays a second screen generated by a second application superimposed on a first screen generated by a first application, comprising:
a step of obtaining a first image displayed on the first screen by an overall image obtaining unit;
a second image acquisition unit acquiring a second image displayed on the second screen;
a display processing unit displaying the second image superimposed on the first image;
an operation processing unit accepting an edit to the second image;
a handwriting data processor converting pen or hand handwriting data into a handwriting data image;
The handwritten data image and the input image input from the outside are superimposed and displayed on the first screen,
When an operation to acquire the first image is input,
an input image acquiring unit acquires the input image, the second image acquiring unit acquires the second image,
The handwritten data processing unit arranges the second image in the handwritten data image,
The image display method , wherein the display processing unit superimposes and displays the handwritten data image in which the second image is arranged and the input image . A display device that displays a second screen generated by a second application superimposed on a first screen generated by a first application,
a whole image acquisition unit that acquires the first image displayed on the first screen;
a second image acquisition unit that acquires a second image displayed on the second screen;
a display processing unit that displays the second image superimposed on the first image;
an operation processing unit that receives edits to the second image;
Function as a handwritten data processing unit that converts handwritten data by pen or hand into a handwritten data image ,
The handwritten data image and the input image input from the outside are superimposed and displayed on the first screen,
When an operation to acquire the first image is input,
an input image acquiring unit acquires the input image, the second image acquiring unit acquires the second image,
The handwritten data processing unit arranges the second image in the handwritten data image,
The display processing unit is a program for displaying the handwritten data image in which the second image is arranged and the input image in an overlapping manner .

Images