JP7342501B2 - Display device, display method, program - Google Patents

Description

The present invention relates to a display device, a display method, and a program.

2. Description of the Related Art Display devices that display data handwritten with a pen or finger on a touch panel are known. An external device can be connected to the display device, and images output from the external device can be displayed. The user can handwrite data over or around the video output by the external device. Video and handwritten data can be saved page by page (one screen at a time), and the image data of the captured video and handwritten data can be displayed again by the user later.

It is also possible to connect a plurality of external devices to the display device, and in this case, the display device simultaneously displays a plurality of images in a plurality of areas created by dividing one screen. The user can handwrite on the area or around the area. 2. Description of the Related Art A technique related to a method of displaying handwritten data when handwritten in a plurality of areas is known (see, for example, Patent Document 1). Patent Document 1 discloses a technology that allows handwritten data to be displayed without being cut off by adjusting the position and size of the writing when one document is enlarged and displayed with handwriting spanning multiple documents. Disclosed.

However, the conventional technology has a problem in that the user must select which area to save when multiple areas are displayed. That is, when a display device to which a plurality of external devices are connected divides its screen and simultaneously displays images of the plurality of external devices and stores the video and handwritten data, the following pattern can be considered.
(i) Save images from external devices one by one as image data.
(ii) All the images displayed on the screen (display) of the display device are saved together as one image data.

Therefore, it would be possible for the user to consider which of (i) and (ii) above is more suitable for storing video or handwritten data, and then allow the user to select a pattern, but this would require a lot of effort for the user to select. There's a problem.

SUMMARY OF THE INVENTION In view of the above-mentioned problems, it is an object of the present invention to provide a display device and the like that can store video of an appropriate area without the user having to select it.

In view of the above-mentioned problems, the present invention provides a display device that displays images input from a plurality of external devices on different areas of a screen, the display device having a touch screen that receives data input based on the contact position of an input means with respect to the screen. When storing images input from a detection unit and a plurality of external devices, the area where the data exists is set as a storage range based on the position of the data received by the contact detection unit and the position of the area. The storage range determination unit includes a storage range determination unit that determines the storage range, and a data storage unit that acquires and stores the image data of the storage range determined by the storage range determination unit from the screen, and the storage range determination unit If the data exists in only one area, only the area where the data exists is determined as the storage range; if the data exists in multiple areas, each area where the data exists is determined as the storage range. If the data exists outside the area, the area where the data exists and the circumscribing rectangle of the data are determined as the storage range, and if the data does not exist on the screen, all the multiple areas are The method is characterized in that the storage range is determined .

It is possible to provide a display device or the like that can save an image of an appropriate area without the user having to select it.

FIG. 3 is a diagram illustrating how a display device stores video and handwritten data. 1 is an example of an overall configuration diagram of an image processing system. 1 is an example of a hardware configuration diagram of a display device. 1 is an example of a functional block diagram of a display device. It is an example of a block diagram of each image layer. FIG. 2 is an example of a functional block diagram showing functions of the display device 2 related to saving handwritten data and area screens when a plurality of videos are input. FIG. FIG. 7 is an example of a flowchart diagram illustrating a method of determining the range of the screen to be saved based on how handwritten data overlaps with the area. FIG. FIG. 3 is a diagram illustrating at least one of a first area and a second area where handwritten data exists and a circumscribed rectangle of the handwritten data. It is an example of a functional block diagram showing the function of the display device 2 that saves handwritten data and area screens for each pattern when a plurality of images are input, and displays them again. FIG. 3 is an example of a flowchart diagram illustrating a method of determining image data to be displayed based on how handwritten data overlaps with an area. FIG. FIG. 3 is an example of a diagram illustrating a method of displaying a circumscribed rectangle including a first area and handwritten data; FIG. FIG. 3 is an example of a flowchart diagram illustrating a method of determining image data to be displayed based on how handwritten data overlaps with an area. FIG. FIG. 6 is a diagram illustrating an example in which a stroke straddles the boundary between the first region and the second region. FIG. 3 is a conceptual diagram showing stroke array data and coordinate array data. This is an example of a setting screen for conditions 1 and 2 displayed on a display device. 1 is a diagram illustrating a configuration example of a display device using a projector. 1 is an example of a hardware configuration diagram of a projector. FIG. 2 is a functional block diagram of each part included in a projector-type display device. FIG. 2 is an example of a functional block diagram showing functions of a display device related to determining a storage range based on how handwritten data overlaps with an area. FIG. FIG. 6 is a diagram illustrating an example of a screen display when video inputs from a plurality of external devices are displayed.

<Display method of display device of this embodiment>
In the display device of this embodiment, when a display device connected to a plurality of external devices splits the screen and simultaneously displays images of the plurality of external devices and saves the video and handwritten data, the handwritten data becomes an area. The storage range is determined based on how they overlap.

FIG. 1 is a diagram illustrating how the display device 2 of this embodiment stores video and handwritten data 9. In the explanation of FIG. 1, an external device 6a and an external device 6b are connected to the display device 2, and the screen is divided into two areas. A video input from the external device 6a (Screen A) is displayed in a first area 101 on the left side of the screen, and a video input from the external device 6b (Screen B) is displayed in a second area 102 on the right side. There is.

In FIG. 1A, handwritten data 9 exists only in the first area 101 where the video input from the external device 6a is displayed. In this case, the display device 2 stores the video and handwritten data 9 in the first area 101 while maintaining the relative position and relative size of the handwritten data 9 with respect to the video input from the external device 6a. For example, the handwritten data 9 overlapping the first area 101 is saved together with the video of the first area 101. Therefore, only the image of the area where the user is handwriting can be saved.

Note that if the handwritten data 9 is present only in the second area 102 where the video input from the external device 6b is displayed, the relative position and relative size of the handwritten data 9 with respect to the video input from the external device 6b is maintained. The image and handwritten data 9 in the second area 102 are saved as they are.

In FIG. 1B, handwritten data 9 exists in both the first area 101 where the video input from the external device 6a is displayed and the second area 102 where the video input from the external device 6b is displayed. In this case, the display device 2 displays the images of the first area 101 and the second area 102 and the handwritten data 9 while maintaining the relative position and relative size of the handwritten data 9 with respect to the images of the first area 101 and the second area 102. save. For example, the handwritten data 9 is saved together with the images of the first area 101 and the second area 102. Therefore, if there are multiple areas where the user is handwriting, multiple videos can be saved at once.

In FIG. 1(c), a first area 101 in which an image input from an external device 6a is displayed, a second area 102 in which an image input from an external device 6b is displayed, and handwritten data outside the area. There are 9. In this case, the display device 2 displays the handwritten data 9 in the first area 101, the second area 102, and the handwritten data 9 while maintaining the relative position and relative size of the handwritten data 9 with respect to the images in the first area 101 and the second area 102. The circumscribed rectangle 602 containing the . For example, a circumscribed rectangle 602 including the first area 101, the second area 102, and the handwritten data 9 is determined, and an image of the handwritten data 9 within the circumscribed rectangle 602 is saved. Therefore, if there is an area where the user is handwriting, and there is handwritten data 9 outside the area, the video and handwritten data 9 can be saved together.

In this way, the display device 2 of this embodiment is
(i) Determine whether handwritten data 9 exists within the area and save only the image of the area where handwritten data 9 exists.
(ii) If handwritten data 9 exists in multiple areas, images of each area are saved together.
(iii) If there is handwritten data 9 outside the area, the video area and the circumscribed rectangle of the handwritten data 9 are saved.
(iv) If handwritten data 9 does not exist within the area, all images are saved together.

In other words, the display device 2 determines the area where handwritten data exists as the storage range. By doing so, the display device 2 of the present embodiment saves the user's effort to select which video to save when the screen is divided to display videos from multiple external devices at the same time and the video is saved. can be reduced.

<About terms>
The handwritten data 9 is data in which a sequence of coordinate points that the user continuously moves the input means on the display is displayed as a locus. Further, a series of operations in which the user presses the input means against the display, continuously moves it, and then releases it from the display is called a stroke, and data handwritten by a stroke is called stroke data. Handwritten data 9 includes one or more stroke data.

The handwritten data 9 includes not only the handwritten data 9 itself, but also text data converted by character recognition of the handwritten data 9, fixed characters such as "Done", stamps displayed as marks, and figures such as circles and stars. , straight lines, etc., may also include data displayed based on user operations.

<System overview>
FIG. 2 is an overall configuration diagram of the image processing system 1 of this embodiment. In addition, in FIG. 2, in order to simplify the explanation, only two display devices 2a, 2b and accompanying electronic pens 4a, 4b, etc. are shown, and three or more display devices 2 and An electronic pen or the like may also be used.
As shown in FIG. 2, the image processing system 1 includes a plurality of display devices 2a and 2b, a plurality of electronic pens 4a and 4b, USB memories 5a and 5b, external devices 6a and 6b, and a television (video) conference terminal. 7a, 7b, and a PC (Personal Computer) 8. Further, the display devices 2a, 2b and the PC 8 are communicably connected via a communication network N. Further, the plurality of display devices 2a and 2b are provided with displays 3a and 3b, respectively.

Further, the display device 2a is configured to display an image drawn by an event caused by the electronic pen 4a (a touch of the tip of the electronic pen 4a or the butt of the electronic pen 4a on the display 3a) on the display 3a. I can do it. Note that the image displayed on the display 3a can be changed based not only on the electronic pen 4a but also on events (gestures such as enlargement, reduction, page turning, etc.) caused by the user's hand Ha or the like.

Further, a USB memory 5a can be connected to the display device 2a, and the display device 2a can read electronic files such as PDF from the USB memory 5a, and the display device 2a can record electronic files in the USB memory 5a. I can do it. The display device 2a is also equipped with a cable 10a1 capable of communication according to standards such as DisplayPort (registered trademark), DVI (Digital Visual Interface), HDMI (registered trademark. High-Definition Multimedia Interface), and VGA (Video Graphics Array). An external device 6a is connected thereto. The cable 10a1 may be a network.

Then, the display device 2a generates an event by touching the display 3a, and transmits event information indicating this event to the external device 6a in the same way as an event from an input device such as a mouse or a keyboard. Similarly, a television (video) conference terminal 7a is connected to the display device 2a via a cable 10a2 capable of communication according to the above standard. Note that the external device 6a and the video conference terminal 7a may communicate with the display device 2a by wireless communication based on various wireless communication protocols such as Bluetooth (registered trademark).

On the other hand, at other bases where the display device 2b is installed, the display device 2b equipped with the display 3b, the electronic pen 4b, the USB memory 5b, the external device 6b, the video conference terminal 7b, the cable 10b1, the cable 10b2 is used. Furthermore, the image displayed on the display 3b can be changed based on an event caused by the user's hand Hb or the like.

As a result, an image drawn on the display 3a of the display device 2a at one location is also displayed on the display 3b of the display device 2b at another location, and conversely, an image drawn on the display 3b of the display device 2b at another location. The image drawn in is displayed on the display 3a of the display device 2a at one location. In this way, the image processing system 1 can perform remote sharing processing in which the same image is shared in a remote location, so it is very convenient to use it for a conference or the like in a remote location.

In addition, below, when indicating arbitrary display devices 2 among the plurality of display devices 2, it will be referred to as "display device 2." When indicating an arbitrary display among a plurality of displays, it is indicated as "display 3." When indicating an arbitrary electronic pen among a plurality of electronic pens, it is indicated as "electronic pen 4." When indicating an arbitrary USB memory among a plurality of USB memories, it is indicated as "USB memory 5." When indicating an arbitrary external device among a plurality of external devices, it is indicated as "external device 6." When indicating an arbitrary teleconference terminal among the plurality of teleconference terminals, it is indicated as "teleconference terminal 7." Further, when indicating an arbitrary hand among a plurality of users' hands, it is indicated as "hand H". When indicating an arbitrary cable among a plurality of cables, it is indicated as "cable 10."

Further, in this embodiment, the display device 2 will be described as an example of an information processing device, but the display device 2 is not limited to this, and other examples of the information processing device include electronic signage (digital signage), sports, weather forecast, etc. It may also be a telestrator used in , a remote image (video) diagnostic device, or the like. A notebook PC is an example of the external device 6, but examples of the external device 6 include a desktop PC, a tablet PC, a smartphone, a PDA (Personal Digital Assistant), a digital video camera, a digital camera, a game console, etc. It may be a terminal that can supply frames. Note that in these information processing devices, a predetermined application (program) for the display device 2 is running, and the application displays the handwritten data 9 based on the contact position on the touch panel.

Furthermore, the communication network N includes the Internet, a LAN (Local Area Network), a mobile phone communication network, and the like. Further, in this embodiment, a USB memory will be described as an example of a recording medium, but the present invention is not limited to this, and various recording media such as an SD card (registered trademark) may be used as other examples of the recording medium. good.

<Hardware configuration of display device>
FIG. 3 is a hardware configuration diagram of the display device 2. As shown in FIG. As shown in FIG. 3, the display device 2 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, an SSD (Solid State Drive) 204, a network I/ F205 and an external device connection I/F (Interface) 206.

Among these, the CPU 201 controls the operation of the display device 2 as a whole. The ROM 202 stores programs used to drive the CPU 201, such as the CPU 201 and an IPL (Initial Program Loader). RAM 203 is used as a work area for CPU 201. The SSD 204 stores various data such as programs for the display device 2. Network I/F 205 controls communication with communication network N. External device connection I/F 206 is an interface for connecting various external devices. The external devices in this case are, for example, a USB (Universal Serial Bus) memory 5 and external devices (microphone 240, speaker 250, camera 260).

The display device 2 also includes a capture device 211, a GPU 212, a display controller 213, a contact sensor 214, a sensor controller 215, an electronic pen controller 216, a short-range communication circuit 219, an antenna 219a of the short-range communication circuit 219, a power switch 222, and Selection switches 223 are provided.

Among these, the capture device 211 causes the display 3 to display video information displayed on the display 3 by the external device 6 as a still image or a moving image.

A GPU (Graphics Processing Unit) 212 is a semiconductor chip that specializes in graphics. The display controller 213 controls and manages screen display in order to output the output image from the GPU 212 to the display 3 or the like. The contact sensor 214 detects when the electronic pen 4, the user's hand H, or the like comes into contact with the display 3.

Sensor controller 215 controls processing of contact sensor 214. The contact sensor 214 performs coordinate input and coordinate detection using an infrared cutoff method. This method of inputting and detecting coordinates is such that two light receiving and emitting devices installed at both ends of the upper side of the display 3 emit multiple infrared rays in parallel to the display 3. This is a method of receiving light that is reflected by a reflecting member and returns along the same optical path as the light emitted by the light receiving element.

The contact sensor 214 outputs the ID of the infrared rays emitted by the two light receiving and emitting devices blocked by the object to the sensor controller 215, and the sensor controller 215 identifies the coordinate position that is the contact position of the object. The electronic pen controller 216 communicates with the electronic pen 4 to determine whether the display 3 has been touched with the tip of the pen or the end of the pen. The near field communication circuit 219 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark).

The power switch 222 is a switch for switching the power of the display device 2 on and off. The selection switches 223 are, for example, a group of switches for adjusting the brightness, shade, etc. of the display on the display 3.

Furthermore, the display device 2 includes a bus line 210. The bus line 210 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 201 shown in FIG. 3.

Note that the contact sensor 214 is not limited to the infrared cutoff type, but can also be a capacitive type touch panel that identifies the touch position by detecting changes in capacitance, or a touch sensor that identifies the touch position by voltage changes between two opposing resistive films. Various detection means may be used, such as a resistive film type touch panel that detects electromagnetic induction generated when a touch object contacts the display section 29, and an electromagnetic induction type touch panel that specifies the contact position. Furthermore, the electronic pen controller 216 may determine whether or not a portion of the electronic pen 4 that the user grasps or other portions of the electronic pen is touched, in addition to the pen tip and pen tail of the electronic pen 4.

The camera 260 is used to capture images of participants during a video conference, but is also used to capture images of not only the participants but also people around the display device 2 . The camera 260 does not need to be an imaging device, and may be a sensor capable of detecting a person, such as an infrared sensor. Further, the camera 260 may be a spherical camera capable of capturing images of the surrounding area at 360 degrees. Cameras with a normal field of view have a relatively narrow field of view, which can make it difficult to detect people, but spherical cameras can image people over a wide range, making it easier to detect people around them. Camera 260 is used as a human detection sensor.

<Functional configuration of display device>
Next, the functional configuration of the display device 2 will be explained using FIGS. 4 and 5. Note that first, the overall functional configuration of the display device 2 will be explained using FIG. 4. FIG. 4 is a functional block diagram of the display device 2. As shown in FIG.

The display device 2 has the functional configuration shown in FIG. 4 using the hardware configuration and program shown in FIG. The display device 2 can be a "host device" that starts the remote sharing process first, and can also be a "participating device" that later participates in the remote sharing process that has already started. Furthermore, the display device 2 is broadly divided into both a client section 20 and a server section 90.

The client section 20 and the server section 90 are functions realized within one housing of the display device 2. When the display device 2 serves as a host device, the display device 2 implements a client section 20 and a server section 90. Further, when the display device 2 becomes a participating device, the client unit 20 is implemented in the display device 2, but the server unit 90 is not implemented. That is, in FIG. 2, when the display device 2a is the host device and the display device 2b is the participating device, the client unit 20 of the display device 2a communicates with others via the server unit 90 implemented within the same display device 2a. It communicates with the client section 20 of the display device 2b. On the other hand, the client section 20 of the display device 2b communicates with the client section of the other display device 2a via the server section 90 implemented in the other display device 2a.

[Functional configuration of client section]
Next, the functional configuration of the client unit 20 will be explained using mainly FIGS. 4 and 5. The client unit 20 includes a video acquisition unit 21 , a coordinate detection unit 22 , an automatic adjustment unit 23 , a contact detection unit 24 , an event distribution unit 25 , an operation processing unit 26 , a gesture processing unit 27 , a video superimposition unit 28 , and an image processing unit 30 , and a communication control section 60.

Among these, the video acquisition unit 21 acquires an output image (corresponding to the video in FIG. 1) of the external device 6 connected to the cable 10. When the video acquisition unit 21 receives the video from the external device 6, it analyzes this video and determines the resolution of the image frame that is the display image of the external device 6 formed by this video, the update frequency of this image frame, etc. Image information is derived and output to the image acquisition section 31.

The coordinate detection unit 22 detects the coordinate position on the display 3 that the user touches with the input means. The trajectory of the user's continuous movement of the input means on the display 3 is handwritten data 9. Further, this handwritten data 9 may be referred to as a stroke. Note that the input means may be either the electronic pen 4 or the user's hand H. The coordinate detection unit 22 also detects the touched area.

The automatic adjustment section 23 is activated when the display device 2 is started (also referred to as when restarted), and detects coordinates using an optical sensor method so that the coordinate detection section 22 can output appropriate values. The detection unit 22 adjusts parameters when processing images from the sensor camera.

The contact detection unit 24 detects an event caused by the user (such as an operation in which the tip of the electronic pen 4 or the pen tail of the electronic pen 4 is pressed (touched) on the display 3).

The event sorting section 25 sorts the coordinate position of the event detected by the coordinate detecting section 22 and the detection result detected by the contact detecting section 24 into events of stroke drawing, UI operation, and gesture operation. Here, "stroke drawing" means that when a stroke image (B) shown in FIG. 5 and described later is displayed on the display 3, the user presses down the electronic pen 4 on the display 3, and this press This is an event in which the electronic pen 4 is moved in this state until it is finally released from the display 3. By this stroke drawing, for example, the alphabet "S", "T", etc. are drawn on the display 3. Note that this "stroke drawing" includes not only drawing an image, but also events such as deleting an already drawn image and editing a drawn image.

“UI operation” is an event in which the user presses a predetermined position with the electronic pen 4 or hand H when the UI image (A) shown in FIG. 5 and described later is displayed on the display 3. . Through this UI operation, for example, the color and width of the line drawn by the electronic pen 4 are set.

“Gesture operation” is an event in which the user touches or moves the display 3 with the hand H when the stroke image (B) shown in FIG. 5 and described later is displayed on the display 3. be. With this gesture operation, for example, by touching the display 3 with the hand H and moving the hand H, the user can enlarge (or reduce) the image, change the display area, or switch pages. can.

The operation processing unit 26 executes various operations based on the UI element determined to be a UI operation by the event distribution unit 25, according to the UI element in which the event occurred. Examples of the elements of this UI include buttons, lists, check boxes, and text boxes. The gesture processing unit 27 executes an operation corresponding to the gesture operation determined by the event distribution unit 25.

The video superimposing unit 28 displays an image superimposed by a display superimposing unit 36 (described later) as a video on the display unit 29. The display section 29 shows display functions realized by the display 3. Further, the video superimposing unit 28 performs picture-in-picture on the video sent from the video conference terminal 7 or the like on the video from the external device 6 or the like. Further, the video superimposing unit 28 performs switching so that the video that has been picture-in-pictured and displayed on a part of the display unit 29 is displayed on the entire display unit 29.

The image processing unit 30 performs superimposition processing of each image layer as shown in FIG. 5, and the like. The image processing section 30 includes an image acquisition section 31, a stroke processing section 32, a UI image generation section 33, a background generation section 34, a layout management section 35, a display superimposition section 36, a page processing section 37, a file processing section 40, and a page data processing section 37. It has a storage unit 300 and a remote license management table 310.

Among these, the image acquisition unit 31 acquires each frame as an image from the video acquired by the video acquisition unit 21. The image acquisition unit 31 outputs this image data to the page processing unit 37. This image corresponds to the output image (C) from the external device 6 etc. shown in FIG. 5. In this embodiment, the output image (C) is referred to as a video.

The stroke processing unit 32 draws an image (handwritten data 9), deletes a drawn image, or edits a drawn image based on events related to stroke drawing allocated by the event distribution unit 25. do. The image created by this stroke drawing corresponds to the stroke image (B) shown in FIG. Further, the results of image drawing, deletion, and editing based on this stroke drawing are stored in the data management unit 80 as operation data to be described later.

The UI image generation unit 33 generates a UI (user interface) image that is preset on the display device 2. This UI image corresponds to the UI image (A) shown in FIG. 5.

The background generation unit 34 receives media data from the page data read out from the page data storage unit 300 by the page processing unit 37 . The background generation unit 34 outputs the received media data to the display superimposition unit 36. Further, the image based on this media data corresponds to the background image (D) shown in FIG. 5. The pattern of the background image (D) is a plain color, a grid display, or the like.

The layout management unit 35 sends layout information indicating the layout of each image output from the image acquisition unit 31, stroke processing unit 32, and UI image generation unit 33 (or background generation unit 34) to the display superimposition unit 36. Managed. Thereby, the layout management unit 35 instructs the display superimposition unit 36 to determine where in the UI image (A) and background image (D) the output image (C) and the stroke image (B) are to be displayed. You can instruct whether to hide it.

The display superimposition unit 36 creates the layout of each image output from the image acquisition unit 31, stroke processing unit 32, and UI image generation unit 33 (background generation unit 34) based on the layout information output from the layout management unit 35. conduct. Furthermore, the display superimposition unit 36 determines the storage range of the video input from the external device 6a or 6b based on how the handwritten data 9 overlaps with each area of the video.

The page processing unit 37 can save the handwritten data 9 as stroke array data (coordinate point sequence) separately from the video, or can also save the handwritten data 9 together with the video as image data. Saving as image data means to generate one piece of image data in which the video and handwritten data 9 are overlapped (synthesized). In this embodiment, for convenience of explanation, a case will be described in which a single piece of image data in which a video and handwritten data 9 are overlapped is generated.

When saving the handwritten data 9 as stroke array data (coordinate point sequence) separately from the video, the page processing unit 37 combines the data of the stroke image (B) and the data of the output image (C) into one page data. and stored in the page data storage unit 300. The data of the stroke image (B) forms part of the page data as stroke array data (each stroke data). The data of the output image (C) forms part of the page data as media data. When this media data is read from the page data storage unit 300, it is treated as background image (D) data.

In addition, the page processing unit 37 sends the media data of the page data once stored to the display superimposition unit 36 via the background generation unit 34, so that the video superimposition unit 28 displays the background image (D). 3 can be redisplayed. Further, the page processing unit 37 can return the stroke arrangement data (each stroke data) of the page data to the stroke processing unit 32, thereby making it possible to re-edit the strokes. Furthermore, the page processing unit 37 can also delete or copy page data.

That is, the data of the output image (C) displayed on the display 3 at the time when the page processing unit 37 stores the page data in the page data storage unit 300 is temporarily stored in the page data storage unit 300, and then When read from the page data storage unit 300, it is read as media data indicating the background image (D). Then, the page processing section 37 outputs stroke array data indicating the stroke image (B) among the page data read from the page data storage section 300 to the stroke processing section 32. Furthermore, the page processing unit 37 outputs media data indicating the background image (D) out of the page data read from the page data storage unit 300 to the background generation unit 34.

The display superimposition unit 36 outputs an output image (C) from the image acquisition unit 31 , a stroke image (B) from the stroke processing unit 32 , a UI image (A) from the UI image generation unit 33 , and an output image (A) from the background generation unit 34 . The background image (D) is superimposed according to the layout specified by the layout management section 35. As a result, as shown in FIG. 5, even if the images overlap, the UI image (A), stroke image (B), output image (C), and background image (D) are displayed in the order in which the user sees them. It is composed of Leia.

Further, the display superimposing unit 36 can also switch between image (C) and image (D) shown in FIG. 5 and superimpose them exclusively on image (A) and image (B). . For example, if the cable 10 between the display device 2 and the external device 6 is disconnected while images (A), (B), and (C) are initially being displayed, the layout management section By specifying 35, image (C) can be removed from the superimposition target and image (D) can be displayed. In this case, the display superimposition unit 36 also performs display enlargement, display reduction, and display area movement processing.

[Functional configuration of server section]
The server section 90 is provided in each display device 2, and any display device 2 can serve as a server section. Therefore, the server section 90 includes a communication control section 70 and a data management section 80.

The communication control unit 70 controls communication with the communication control unit 70 in the client unit 20 in the same display device 2 and with the communication control unit 70 in the client unit 20 in another display device 2 via the communication network N. The data management unit 80 manages operation data, image data, and the like.

<Display device functions related to screen reconfiguration according to video input>
FIG. 6 is an example of a functional block diagram showing the functions of the display device 2 related to saving handwritten data and area screens when a plurality of videos are input. FIG. 6 shows the image acquisition section 31, stroke processing section 32, display superimposition section 36, and page processing section 37 shown in FIG. 4.

The image acquisition section 31 receives the video acquired by the video acquisition section 21 at each of the input terminals 1 and 2. In FIG. 6, the number of input terminals is two. The input terminal is a terminal to which the cable 10a1, such as the HDMI (registered trademark) described above, is connected. External devices 6 that output video are connected to these input terminals, respectively. Therefore, the image acquisition unit 31 receives video input in real time equal to the number of external devices 6, and transmits the video for each input terminal to the display superimposition unit 36 in real time.

The display superimposing section 36 includes a video receiving section 11 , a stroke receiving section 12 , a screen combining section 13 , and a storage range determining section 14 . The video receiving unit 11 receives videos for each input terminal in real time from the image acquiring unit 31 for the number of external devices 6. The stroke receiving unit 12 acquires the handwritten data 9 from the stroke processing unit 32 and holds it in units of pages (the entire screen of the display).

The video receiving section 11 transmits each video to the screen composition section 13. Stroke receiving section 12 transmits handwritten data 9 to screen composition section 13 .

The screen synthesis unit 13 synthesizes the video from each external device 6 and the handwritten data 9, and displays the synthesized image on one display. The screen composition unit 13 divides the screen according to the number of images (number of divisions). It is assumed that the correspondence between the number of images and the coordinates of each area is determined in advance. Even if the coordinates are not fixed, the coordinates of each area are known to the screen synthesis unit 13.
Number of images 1 → full screen (no division)
Number of videos: 2 → divided into 2 Number of videos: 3 → divided into 3:
Number of images N→N division The screen synthesis unit 13 synthesizes the stroke image (B) with the output image (C) of the number of areas (number of divisions) determined in this way. Combining in this case simply means displaying them in an overlapping manner.

When the storage range determining unit 14 receives a notification from the operation processing unit 26 that a screen storage operation by the user has been detected, the storage range determining unit 14 determines the range to be stored depending on how the handwritten data 9 overlaps with the area. Simply put, the storage range determination unit 14 determines the area where handwritten data exists as the storage range. Note that the storage range determination unit 14 holds the coordinates of the area for each input terminal acquired from the screen composition unit 13.

The rules by which the storage range determination unit 14 determines the storage range are as follows.
(i) Determine whether handwritten data 9 exists within the area and save only the image of the area where handwritten data 9 exists.
(ii) If handwritten data 9 exists in multiple areas, images of each area are saved together.
(iii) If there is handwritten data 9 outside the area, the video area and the circumscribed rectangle of the handwritten data 9 are saved.
(iv) If handwritten data 9 does not exist within the area, all images are saved together.

Saving refers to capturing or capturing screen data. This refers to saving a storage range of the image data in which the output image (C) and the stroke image (B) synthesized by the screen synthesis unit 13 are synthesized as electronic data.

The storage range (coordinates of the upper left corner and the upper right corner) determined by the storage range determination unit 14 is sent to the page processing unit 37. The page processing section 37 has a data storage section 15. The data storage unit 15 does not store the video and the handwritten data 9 separately as described above, but stores image data in which the output image (C) and the stroke image (B) are combined. However, they may be saved separately. Further, the handwritten data 9 itself is stored separately by the page processing unit 37.

<Operating procedure>
Next, with reference to FIG. 7, the operation of the display device 2 when a user performs an operation to save a screen when a plurality of videos are input will be described. FIG. 7 is an example of a flowchart illustrating a method of determining the range of the screen to be saved depending on how the handwritten data 9 overlaps with the area. In the description of FIG. 7, it will be assumed that the screen of the display device 2 is divided into two areas. The external device 6 that inputs an image to the first area 101 is an external device 6a, and the external device 6 that inputs an image to the second area 102 is an external device 6b.

The operation processing unit 26 detects an operation to save the screen in a state where the screen is divided into two areas and the video input from the external device 6 is displayed in each area (S1). The data may be saved automatically (for example, periodically) without being triggered by a user's operation.

Thereby, the storage range determination unit 14 determines whether handwritten data 9 exists within the screen regardless of the area (S2). The storage range determination unit 14 determines whether the screen synthesis unit 13 is displaying the handwritten data 9 on the screen, regardless of the coordinates of the handwritten data 9 or the coordinates of the area. That is, it is sufficient to determine whether there is handwritten data 9 on the screen.

If the determination in step S2 is No, if there is no handwritten data 9, the storage range determining unit 14 determines to save the first area 101 and the second area 102 (S3). Since the storage range determination unit 14 sends the coordinates of the storage range including the first area 101 and the second area 102 to the data storage unit 15, the data storage unit 15 is specified by the coordinates of the first area 101 and the second area 102. Save the image within the range.

If the determination in step S2 is Yes, the handwritten data 9 exists, so the storage range determining unit 14 determines whether the handwritten data 9 exists only in the first area 101 where the video input from the external device 6a is displayed. makes a judgment (S4). Since the storage range determining unit 14 holds the coordinates of the area for each input terminal, it determines the presence or absence of the handwritten data 9 for each area.

If all of the handwritten data 9 falls within the first area 101 where the video input from the external device 6a was displayed (Yes in S4), the storage range determining unit 14 determines to save only the first area 101 ( S5). Since the storage range determination unit 14 sends the coordinates of the first area 101 to the data storage unit 15, the data storage unit 15 stores the image data of the video and handwritten data in the range specified by the coordinates of the first area 101. Thereby, the relative position and relative size of the handwritten data 9 with respect to the video can be maintained (see FIG. 1(a)).

If all of the handwritten data 9 does not fit in the first area 101 where the video input from the external device 6a is displayed (No in S4), the second area 102 where the video input from the external device 6b is displayed. The storage range determination unit 14 determines whether or not the handwritten data 9 exists only within (S6). Since the storage range determining unit 14 holds the coordinates of the area for each input terminal, it determines the presence or absence of the handwritten data 9 for each area.

If all of the handwritten data 9 is within the second area 102 where the video input from the external device 6b is displayed (Yes in S6), the storage range determining unit 14 determines to save only the second area 102 ( S7). Since the storage range determining unit 14 sends the coordinates of the second area 102 to the data storage unit 15, the data storage unit 15 stores the image data of the video and handwritten data in the range specified by the coordinates of the second area 102. Thereby, the relative position and relative size of the handwritten data 9 with respect to the video can be maintained.

If all of the handwritten data 9 does not fit within the second area 102 where the video input from the external device 6b is displayed (No in S6), the handwritten data 9 exists only within the first area 101 and the second area 102. The storage range determining unit 14 determines whether or not to do so (S8). Since the storage range determination unit 14 holds the coordinates of the area for each input terminal, it determines the presence or absence of the handwritten data 9 for each area.

If the handwritten data 9 exists only in the first area 101 and the second area 102 (Yes in S8), the storage range determining unit 14 determines to save the first area 101 and the second area 102 (S9). Since the storage range determination unit 14 sends the coordinates of the first area 101 and the second area 102 to the data storage unit 15, the data storage unit 15 stores the image in the range specified by the coordinates of the first area 101 and the second area 102. and the image data of the handwritten data (see FIG. 1(b)). Thereby, the relative position and relative size of the handwritten data 9 with respect to the video can be maintained.

If the handwritten data 9 exists in areas other than the first area 101 or the second area 102 (No in S8), the handwritten data 9 exists not only in the first area 101 or only in the second area 102, so it can be saved. The range determining unit 14 determines to save at least one of the first area 101 and the second area 102 where the handwritten data 9 exists and the circumscribed rectangle of the handwritten data 9 (S10). How to obtain the circumscribed rectangle will be explained with reference to FIG. The storage range determination unit 14 sends the coordinates of the circumscribed rectangle determined by the coordinates of at least one of the first area 101 or the second area 102 and the coordinates of the handwritten data 9 to the data storage unit 15. The image data of the video and handwritten data in the range specified by the coordinates of the rectangle is saved (see FIG. 1(c)). Thereby, the relative position and relative size of the handwritten data 9 with respect to the video can be maintained.

Through the above processing, when the display device 2 to which a plurality of external devices 6 are connected splits the screen and saves the video and handwritten data 9 while simultaneously displaying the video of a plurality of external devices, the handwritten data 9 is The storage range can be determined based on how it overlaps. Therefore, the user's effort in selecting which video to save can be reduced.

<At least one of the first area and the second area where handwritten data exists and the circumscribing rectangle of the handwritten data>
FIG. 8 is a diagram illustrating at least one of the first area 101 and the second area 102 in which the handwritten data 9 exists and the circumscribed rectangle of the handwritten data 9.

First, in FIG. 8A, handwritten data 9 exists in the first area 101 and outside the area. Therefore, the storage range determining unit 14 determines the circumscribed rectangle 601 of the handwritten data 9, and determines the first area 101 and the circumscribed rectangle 602 of the circumscribed rectangle 601 as the storage range. Note that such a circumscribed rectangle 602 can be specified by determining the rightmost and leftmost coordinates and the uppermost and lowermost coordinates of the first region 101 and the circumscribed rectangle 601.

The same applies to the case shown in FIG. 8(b) because handwritten data 9 exists outside the first area 101 and the area. Due to the circumscribed rectangle 602, in this case, a portion of the second area 102 is also saved.

In FIG. 8C, handwritten data 9 exists in the first area 101, the second area 102, and outside the area. Therefore, the storage range determination unit 14 determines the circumscribed rectangle 601 of the handwritten data 9, and determines the circumscribed rectangles 602 of the three rectangles, the first area 101, the second area 102, and the circumscribed rectangle 601, as the storage range.

Note that the circumscribing rectangle 601 of the handwritten data 9 is set in FIG. 8 for the sake of explanation, and the circumscribing rectangle 602 can be obtained without setting the circumscribing rectangle 601.

<Summary>
As explained above, the display device 2 of the present embodiment saves the user's effort in selecting which video to save when the screen is divided and videos from multiple external devices are displayed at the same time. can be reduced.

In Example 1, only the determined storage range was stored. In this embodiment, images are saved in all the patterns described in the first embodiment regardless of the presence or absence of handwritten data, and the storage range to be displayed is determined when the images are displayed.

By doing so, image data in an appropriate storage range can be displayed depending on how the handwritten data 9 overlaps with the area. That is, if you want to write to a video input from any one external device 6, it is difficult to write because the size of one video becomes small if the image data is one in which all videos are stored together. In this case, in this embodiment, if the handwritten data 9 is only in one area, only the image data of one area is displayed, making it easier to handwrite.

Furthermore, if you want to write by hand across multiple videos, you cannot write to the image data saved for each video. In this case, in this embodiment, if there is no handwritten data 9 on the screen, or if there is handwritten data 9 in multiple areas, the image data in the storage range where multiple areas are saved together is displayed. You can write by hand across multiple videos.

Furthermore, if the image data displayed by the display device 2 is not what the user desires, the user can select image data to be displayed from among the stored image data.

<About functions>
Note that this embodiment will be described on the assumption that the hardware configuration diagram of FIG. 3 described in the above embodiment can be used.

FIG. 9 is an example of a functional block diagram showing the function of the display device 2 that saves handwritten data and area screens for each pattern and displays them again when a plurality of images are input. In this embodiment, the components with the same reference numerals in FIG. 6 perform the same functions, so only the main components of this embodiment will be mainly described in some cases.

The display device 2 of this embodiment has a display control section 16. Furthermore, the function of the storage range determination unit 14 is different from that in the first embodiment. The storage range determining unit 14 of this embodiment determines to save image data for all patterns. That is, it is decided to save the image data in the following four patterns.
- Save only the first area - Save only the second area - Save only the first and second areas - Save at least one of the first and second areas and the circumscribed rectangle of handwritten data 9 If there is handwritten data stores handwritten data and image data including video.
The display control unit 16 displays the stored image data on the display 3 according to the following policy.
(i) If the handwritten data 9 exists only in one area, the image data of that area is displayed.
(ii) When handwritten data 9 exists in multiple areas, image data in which images of each area are stored together is displayed.
(iii) If there is handwritten data 9 outside the area, the area where the handwritten data is located and the image data in which the circumscribed rectangle of the handwritten data 9 are saved are displayed.
(iv) If handwritten data 9 does not exist within the area, image data in which all images are stored together is displayed.

<Operating procedure>
FIG. 10 is an example of a flowchart diagram illustrating a method of determining image data to be displayed based on how handwritten data 9 overlaps with an area. In the description of FIG. 10, differences from FIG. 7 may be explained.

In a state where the screen is divided into two areas and the video input from the external device 6 is displayed in each area, the operation processing unit detects an operation to save the screen (S11).

As a result, the storage range determining unit 14
- Save only the first area - Save only the second area - Save only the first and second areas - Save at least one of the first and second areas and the circumscribed rectangle of handwritten data 9 in four patterns. The range is determined and the storage range (coordinates) is instructed to the data storage unit 15 (S12).

The data storage unit 15 stores image data of only video or handwritten data and image data of video for each pattern. Note that if there is no handwritten data 9 outside the area, a circumscribing rectangle cannot be set, so the fourth step ``save the circumscribing rectangle of handwritten data 9 and at least one of the first area and the second area'' is not performed.

The following determination methods in steps S13, S15, S17, and S19 may be the same as in FIG. 7.
Further, in step S14, the display control unit 16 determines that the first area 101 and the second area 102 display the saved image data (S14). As a result, if the user has not written anything by hand, they can handwrite on either of the multiple images, or even across both.

In step S16, the display control unit 16 determines to display only the first area 101 (S16). This allows the user to enlarge only one area and handwrite.

In step S18, the display control unit 16 determines to display only the second area 102 (S18). This allows the user to enlarge only one area and handwrite.

In step S20, the display control unit 16 determines to display the image data of the first area 101 and the second area 102 (S20). As a result, when the user writes by hand in the first area 101 and the second area 102, the user can display the handwriting again and write on either of the plurality of images, or can write by hand across the plurality of images.

In step S21, the display control unit 16 determines to display image data of at least one of the first area 101 or the second area 102 and the circumscribed rectangle of the handwritten data 9 (S21). Thereby, when the user writes by hand outside the area, at least one of the first area 101 or the second area 102 and the handwritten data 9 can be displayed in a large size.

For example, the display device 2 displays each image data saved in step S12 as a thumbnail, so if the user wants to display an image other than the images displayed in steps S14, S16, S18, S20, and S21, the user can select the image he or she wants to display from the thumbnails. can be selected and displayed.

<Screen display example>
A method for displaying image data including handwritten data 9 will be supplemented with reference to FIG. FIG. 11 is a diagram illustrating a method of displaying the first area 101 and the circumscribed rectangle 104 including the handwritten data 9. The same explanation can be given when only the first area 101, only the second area 102, or the first area 101 and the second area 102 are saved together.

The display control unit 16 first reads the image data of the circumscribed rectangle 104 that includes the first area 101 and the circumscribed rectangle 103 that the data storage unit 15 has saved.

Since the aspect ratio of this circumscribed rectangle 104 is not necessarily the same as that of the entire screen (or display 3), the display control unit 16 may set margins in the vertical or horizontal direction. Considering the situation where there is no handwritten data 9 in the second area 102, the circumscribed rectangle 104 is longer vertically than the entire screen, so margins are often set in the horizontal direction. For example, if the height of the circumscribed rectangle 104 is H1 and the width is W1, W1 is increased by adding a margin so that W1/H1 becomes the same as the aspect ratio of the entire screen.

In FIG. 11A, for convenience of explanation, the height of the circumscribed rectangle 104 with the same aspect ratio is assumed to be H1, and the width thereof is assumed to be W1. Further, the height of the entire screen is assumed to be H2, and the width is assumed to be W2. Therefore, the magnification factor for maximization is R2=H2/H1=W2/W1.

Next, coordinate transformation of the circumscribed rectangle 104 will be explained. The origin (0,0) is the upper left corner of the entire screen, and the coordinates of the upper left corner of the circumscribed rectangle 104 at the time of saving are (E,F). When redisplaying, it is assumed that the coordinates of the upper left corner of the circumscribed rectangle 104 coincide with the origin of the entire screen. Therefore, the coordinates (x1, y1) of an arbitrary point within the circumscribed rectangle 104 are converted into coordinates when the coordinates (E, F) of the upper left corner of the circumscribed rectangle 104 are set as the origin. The coordinates (X1, Y1) after converting (x1, y1) are as follows.
X1=x1-E
Y1=y1-F
By enlarging at the enlargement rate R2 to maximize X1 and Y1, the coordinates (X2, Y2) after the division is canceled are obtained (FIG. 11(b)).
X2=R2・X1
Y2=R2・Y1
The explanation of FIG. 11 is for the case where the saved image data retains the original coordinates, and the explanation for the case where the saved image data does not retain the original coordinates (what kind of circumscribed rectangle) (and the coordinates of the upper left corner are (0, 0)), it is sufficient to simply enlarge at the enlargement rate R2.

<Summary>
According to this embodiment, image data in an appropriate storage range can be displayed depending on how the handwritten data 9 overlaps with the area.

In this example,
(Condition 1)
・Does one stroke span multiple areas?
- Is the difference in writing times of handwritten data 9 in multiple areas within a certain time?
The display device 2 that changes the image data to be displayed depending on the situation will be described.

Furthermore, in this example,
(Condition 2)
- Does the last handwriting exist in the image of the second area 102?
- Is the video of the second area 102 the last one connected and displayed?
- Has the image in the second area 102 changed?
The display device 2 that changes the image data to be displayed depending on the situation will be described.

Condition 1 makes it possible to more accurately determine whether to display the first area 101 and the second area 102 together. Further, in the case where the first area 101 and the second area 102 are not displayed together, it is possible to determine which image data of the first area 101 and the second area 102 should be displayed based on condition 2.

Note that this embodiment will be described on the assumption that the hardware configuration diagram of FIG. 3 and the functional block diagram of FIG. 9 described in the above embodiment can be used.

<Operating procedure>
FIG. 12 is an example of a flowchart diagram illustrating a method of determining image data to be displayed based on how handwritten data 9 overlaps with an area. In the description of FIG. 12, differences from FIG. 10 may be explained.

In FIG. 12, the determination conditions in step S19-2, which corresponds to step S19 in FIG. 10, are different from those in FIG. 10, and the determination in step S22 is added.

First, in step S19-2, the display control unit 16 determines condition 1 in addition to the condition "only in the first area and second area" (S19-2). Condition 1 is as above,
(i) Does one stroke span multiple areas?
(ii) Is the difference in writing times of handwritten data 9 to multiple areas within a certain time?
Either or both.

If the determination in step S19-2 is Yes, the display control unit 16 displays the image data in which the first area 101 and the second area 102 are stored together (S20). If the determination in step S19-2 is No, the process proceeds to step S22.

The determination in (i) will be explained using FIG. 13. In FIG. 13, a stroke 611 straddles a boundary 613 between the first region 101 and the second region 102. In FIG. The display control unit 16 sequentially extracts the coordinate points 612 that constitute the stroke 611, and determines that one stroke 611 straddles multiple regions when the X coordinate of the boundary 613 between the first region 101 and the second region 102 is exceeded. It is determined that

If the handwritten data 9 does not span multiple videos, it is presumed that the user wants to handwrite each video individually. Therefore, as a result of the determination in (i), the image data in which the first area 101 and the second area 102 are stored together is displayed, thereby reducing the user's effort to select another video.

The determination in (ii) will be explained using FIG. 14. FIG. 14(a) shows an example of stroke array data. As shown in FIG. 14, one stroke array data is represented by a plurality of stroke data. One stroke data includes a stroke data ID for identifying this stroke data, a start time indicating the time when writing one stroke starts, an end time indicating the time when writing one stroke ends, a color of the stroke, It shows coordinate array data ID for identifying the width of the stroke and the array of passing points of the stroke.

Furthermore, this coordinate array data shows detailed information as shown in FIG. 14(b). FIG. 14(b) is a conceptual diagram showing coordinate array data. As shown in FIG. 14(b), the coordinate array data consists of one point (X coordinate value, Y coordinate value) on the display, and the time difference from the stroke start time when passing this one point. (ms) and information on the writing pressure of the electronic pen 4 at this one point. That is, the collection of one point shown in FIG. 14(b) is represented by one coordinate array data shown in FIG. 14. For example, when a user draws the alphabet "S" with the electronic pen 4, it is written in one stroke, but since the user passes through multiple passing points before finishing drawing "S", the coordinate array data is information is shown.

The display control unit 16 determines whether the stroke data belongs to the first area 101 or the second area 102 based on the coordinates, and displays the start time or end time of the stroke data in the first area 101 and the second area 102. Calculate the time at which the difference between the start time or end time of the stroke data is the smallest. If this difference is less than the threshold, it is determined that condition 1 is satisfied.

If the time at which the handwritten data 9 was handwritten differs greatly between regions, it is considered that there is a high possibility that the user handwritten the handwritten data 9 for each region. Therefore, as a result of the judgment in (ii), the image data in which the first area 101 and the second area 102 are stored together is displayed to the user who wishes to draw by hand on each video individually, and the user may The time and effort required to select videos can be reduced.

Condition 2 is as above,
(iii) Does the last handwritten note exist in the image of the second area 102?
(iv) Was the video of the second area 102 the last one connected and displayed?
(v) Has the image in the second area 102 changed?
Any one or two or more of the following.

If condition 2 is Yes, the display control unit 16 determines to display the image data of only the second area 102, and if condition 2 is No, the display control unit 16 determines to display the image data of only the first area 101. do.

The determination in (iii) will be explained using FIG. 14. Since each stroke data has a start time or an end time, the display control unit 16 determines whether handwritten data 9 with the latest end time exists in the second area 102. If (iii) holds true, it is considered that the user would like to continue handwriting in the second area 102. If the determination in (iii) is Yes, the display control unit 16 determines to display the second area 102, so that the image data of the second area 102 where the last handwritten data 9 exists can be displayed.

The determination in (iv) is whether the external device 6a displayed in the second area 102 of the two input terminals was connected (displayed) to the input terminal after the external device 6b displayed in the first area 101. This is the judgment. For this reason, the video receiving unit 11 records the order in which the external devices 6 are connected to the input terminals. The user may handwrite on a video input from the external device 6 that is connected later, and when the user writes by hand on the video that was last connected and displayed, the image data in the second area 102 is can be displayed.

The determination (v) is a determination as to whether the image in the second area 102 is changing. If the image has changed, it is meaningless as handwritten data 9 for the image. Therefore, there is little need to display the handwritten data 9 for the second area 102 where the image is changing. Therefore, if the image in the second area 102 is changing, the image data of only the first area 101 is displayed, and if it is not changing, the image data of only the second area 102 is displayed, so that the user can write by hand. Image data of the area can be displayed.

Note that whether or not there has been a change can be determined by calculating the sum of squares of the difference in pixel values of the video of the second region 102 between times t and t+1, and comparing it with a threshold value.

<User settings for condition 1 and condition 2>
As shown in FIG. 15, conditions 1 and 2 in this embodiment may be set by the user. FIG. 15 is an example of a setting screen for conditions 1 and 2 displayed on the display device 2.

As shown in FIG. 15, a check box 630 is displayed at the beginning of each condition 1 and condition 2, and the user can select one or more conditions 1 and one or more conditions 2.

<Summary>
Therefore, according to the present embodiment, it is possible to more accurately determine whether to display the first area 101 and the second area 102 together based on condition 1. Further, when the first area 101 and the second area 102 are not displayed together, it is possible to determine which image data of the first area and the second area should be displayed based on condition 2.

In the first embodiment, a touch panel type display device 2 was explained, but in this embodiment, a projector projection type display device 2 will be explained. Note that an example of a projector projection type display device 2 like this embodiment is described in Patent Document 2.

FIG. 16 is a diagram showing a configuration example of a display device 2 using a projector according to this embodiment. In the display device 2, an external device 6 is connected to a projector 400 installed above the screen SC.

Projector 400 receives image data transmitted from external device 6 as an image output device, and projects an image based on this image data onto screen SC as projection surface. The projector 400 is of a short focus type, is installed directly above the screen SC, and projects an image diagonally downward.

The projector 400 not only projects images based on image data transmitted from the external device 6, but also projects images based on image data stored inside the projector 400 and images generated inside the projector 400 as described below. An image can be projected onto a screen SC.

The projector 400 can project whether the image data received from the external device 6 is still image data or moving image (video) data. Further, the screen SC is not limited to a flat plate fixed to a wall surface, and the wall surface itself can also be used as the screen SC.

The display device 2 can be operated using the input means 402 on the screen SC on which the projector 400 projects an image. The input means 402 is, for example, a pen-shaped device, and the user holds the input means 402 in his hand and uses it by pressing the tip against the screen SC. An operation switch 403 for detecting a pressing operation is provided at the tip of the input means 402, and when the user presses the tip of the input means 402 against the screen SC, the operation switch 403 is turned on. The user performs a position instruction operation by pressing the tip of the input means 402 against the screen SC at an arbitrary position on the screen SC.

The projector 400 has a function of detecting the position of the input means 402 on the screen SC. When the user performs a position instruction operation, the position where the tip of the input means 402 touches the screen SC is detected as the instruction position.

The display device 2 receives a position input operation performed by the user using the input means 402, and reflects this operation on the projected image. Specifically, when the display device 2 detects a position instruction operation performed by the user, it draws handwritten data 9 and the like according to the specified position, and projects the drawn handwritten data 9 on the screen SC. The display device 2 can draw the handwritten data 9 so as to overlap with the projected image while projecting some kind of image onto the screen SC, and starts drawing the image after transitioning to a state in which no image is projected. You can also. In addition, the handwritten data 9 drawn by this function can be saved as image data, and it is also possible to save only the drawn image as image data, and the projection that was projected on the screen SC at the time of drawing. It can also be saved together with the image as a single image data.

Here, the display device 2 has a function of generating an image according to a position indicated by the input means 402 and projecting it by the projector 400, in a "PJ mode" executed by the projector 400, and in a "PC mode" executed by the external device 6. ” are both executable. In the "PJ mode", the projector 400 detects the designated position of the input means 402, generates an image according to the detected designated position, and projects it on the screen SC.

Furthermore, in the "PC mode", the projector 400 performs an operation of projecting an image input from the external device 6 onto the screen SC, and the projector 400 does not perform drawing. In the PC mode, the projector 400 detects the indicated position of the input means 402 and transmits operation data indicating the detected indicated position to the external device 6. The external device 6 that has received this operation data executes drawing to generate handwritten data 9 based on the coordinates of the indicated position, and displays an image of the drawn handwritten data 9 alone or by superimposing it on another image. generate image data. The external device 6 outputs the generated image data for display to the projector 400. Thereby, an image drawn according to the operation of the input means 402 is projected onto the screen SC.

The PJ mode can be executed by operating the projector 400. Further, the PC mode can be executed by switching the projector 400 to the PC mode and having the external device 6 execute a function of controlling the projector 400 and a function of performing drawing. Therefore, switching from PC mode to PJ mode can be performed by operating projector 400, and switching from PJ mode to PC mode can be performed by operating both projector 400 and external device 6.

Further, a tool bar 411 is projected on the screen SC together with the projected image. A plurality of function buttons 412 and the like for causing the projector 400 to execute various functions are arranged on the tool bar 411. When a position overlapping one of the function buttons 412 is specified by operating the input means 402, the projector 400 executes the function assigned to the corresponding function button 412.

<Hardware configuration example>
FIG. 17 is a hardware configuration diagram of the projector 400. As shown in FIG. 17, the projector 400 includes a CPU (Central Processing Unit) 801, a ROM (Read Only Memory) 802, a RAM (Random Access Memory) 803, a media I/F (Interface) 807, an operation unit 808, and a power switch. 809, bus line 810, network I/F 811, LED (Light Emitting Diode) drive circuit 814, LED light source 815, projection device 816, projection lens 817, external device connection I/F (Interface) 818, fan drive circuit 819, cooling A fan 820 is provided.

Of these, the CPU 801 controls the overall operation of the projector 400. The ROM 802 stores programs used to drive the CPU 801. RAM803 is used as a work area for CPU801.

A media I/F 807 controls reading or writing (storage) of data to a recording medium 806 such as a flash memory.

The operation unit 808 is provided with various keys, buttons, LEDs, etc., and is used by the user to perform various operations other than turning on/off the power of the projector 400. For example, the operation unit 808 receives instruction operations such as a projection image size adjustment operation, a color tone adjustment operation, a focus adjustment operation, a keystone adjustment operation, and outputs the received operation contents to the CPU 801.

The power switch 809 is a switch for switching the power of the projector 400 on and off.

The bus line 810 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 801 shown in FIG. 17.

The network I/F 811 is an interface for data communication using a communication network N such as the Internet.

The LED drive circuit 814 controls turning on and off of the LED light source 815 under the control of the CPU 801.

When the LED light source 815 is turned on under the control of the LED drive circuit 814, it irradiates the projection device 816 with projection light.

The projection device 816 modulates the projection light from the LED light source 815 using a spatial light modulation method based on the image data provided via the external device connection I/F 818, and outputs the modulated light through the projection lens 817. , and project it as an image onto the projection surface of the screen. As the projection device 816, for example, a liquid crystal panel or a DMD (Digital Micromirror Device) is used. The LED drive circuit 814, LED light source 815, projection device 816, and projection lens 817 collectively function as a projection unit that projects a projection image onto a projection surface based on image data.

The external device connection I/F 818 is directly connected to a PC, and acquires control signals and image data with the PC.

The fan drive circuit 819 is connected to the CPU 801 and the cooling fan 820, and drives/stops driving the cooling fan 820 based on a control signal from the CPU 801.

By rotating, the cooling fan 820 exhausts the air inside the projector 400 and cools the inside of the projector 400.

Further, when power is supplied, the CPU 801 starts up according to a control program stored in the ROM 802 in advance, and gives a control signal to the LED drive circuit 814 to light the LED light source 815, and also to the fan drive circuit 819. A control signal is given to rotate the cooling fan 820 at a predetermined rated rotation speed. Further, in the projector 400, when the supply of power from the power supply circuit is started, the projection device 816 enters a state capable of displaying an image, and further, power is supplied from the power supply circuit to various other components.

Further, in the projector 400, when the power switch 809 is turned off, a power OFF signal is sent from the power switch 809 to the CPU 801, and when the CPU 801 detects the power OFF signal, it gives a control signal to the LED drive circuit 814 to turn the LED The light source 815 is turned off. Thereafter, when a predetermined period of time has elapsed, the CPU 801 gives a control signal to the fan drive circuit 819 to stop the cooling fan 820, ends its own control processing, and finally gives an instruction to the power supply circuit to reduce the power supply power. supply will be stopped.

<About functions>
FIG. 18 is a functional block diagram of each part of the projector-type display device 2. As shown in FIG. The projector 400 includes an I/F (interface) section 405 that is connected to an external device 6, a video playback device, a DVD playback device, and other external devices. The I/F unit 405 is, for example, a USB interface, a wired or wireless LAN interface, a VGA terminal into which an analog video signal is input, a DVI (Digital Visual Interface) into which digital image data is input, a composite such as NTSC, PAL, SECAM, etc. It is equipped with an S video terminal to which a video signal is input, an RCA terminal to which a composite video signal is input, a D terminal to which a component video signal is input, an HDMI connector compliant with the HDMI (registered trademark) standard, and the like. The projector 400 is connected to the external device 6 via a LAN interface or a USB interface included in the I/F section 405.

The projector 400 is broadly divided into a projection unit 420 that forms an optical image and an image processing system that processes image data. The projection section 420 includes an illumination section 421, a light modulation section 422, and a projection section 423. The illumination unit 421 includes a light source such as a xenon lamp, an ultra-high pressure mercury lamp, an LED (Light Emitting Diode), and a laser light source. Further, the illumination section 421 may include a reflector and an auxiliary reflector that guide the light emitted by the light source to the light modulation section 422, and may include a lens group (not shown), a polarizing plate, or a polarizing plate for improving the optical characteristics of the projected light. It may also include a light control element or the like that reduces the amount of light emitted by the light source on the path to the light modulation section 422.

The light modulator 422 includes three transmissive liquid crystal panels corresponding to the three primary colors of RGB, for example, and generates image light by modulating the light that passes through the liquid crystal panels. The light from the illumination unit 421 is separated into three color lights of RGB, and each color light is incident on each corresponding liquid crystal panel. The colored lights that have passed through each liquid crystal panel and have been modulated are combined by a combining optical system such as a cross dichroic prism, and then emitted to the projection unit 423.

The projection unit 423 includes a zoom lens that enlarges/reduces the image to be projected and adjusts the focus, a zoom adjustment motor that adjusts the degree of zoom, a focus adjustment motor that adjusts the focus, and directs the projection light toward the screen SC. It is equipped with a reflective concave mirror. The projection unit 423 performs zoom adjustment and focus adjustment of the image light modulated by the light modulation unit 422, guides the light that has passed through the lens group toward the screen SC using a concave mirror, and forms an image on the screen SC. The projection unit 420 includes a projection optical system drive unit 416 that drives each motor included in the projection unit 423 under the control of the control unit 430, and a light source drive unit that drives the light source included in the illumination unit 421 under the control of the control unit 430. section 414 is connected. Note that the specific configuration of the projection unit 423 is not limited to the above example, and for example, a configuration that does not use a mirror, including a concave mirror, projects the light modulated by the light modulation unit 422 onto the screen SC using a lens, and forms an image. It is also possible.

On the other hand, the image processing system is mainly configured with a control unit 430 that integrally controls the entire projector 400, and includes a control program 436 executed by the control unit 430, and a storage unit 435 that stores data processed by the control unit 430. , an operation detection unit 417 that detects operations via the operation panel 419 and the remote control light receiving unit 418, an image processing unit 440 that processes an input image input via the I/F unit 405, and an image processed by the image processing unit 440. It includes a light modulation device drive section 415 that drives the light modulation section 422 based on a signal to perform drawing.

The operation detection section 417 is connected to a remote control light receiving section 418 and an operation panel 419. The remote control light receiving section 418 receives an infrared signal transmitted by the remote control used by the user of the projector 400 in response to a button operation. The remote control light receiving section 418 decodes the infrared signal received from the remote controller, generates operation data indicating the operation content on the remote controller, and outputs it to the control section 430.

The operation panel 419 is provided on the exterior casing of the projector 400 and includes various switches and indicator lamps. Under the control of the control unit 430, the operation detection unit 417 lights or blinks the indicator lamp of the operation panel 419 as appropriate depending on the operating state and setting state of the projector 400. When a switch on the operation panel 419 is operated, operation data corresponding to the operated switch is output from the operation detection section 417 to the control section 430.

The control unit 430 outputs the video input from the I/F unit 405 to the image processing unit 440. The image processing unit 440 includes an image input unit 441 that buffers input image data, and performs various conversion processes such as interlace/progressive conversion, resolution conversion, and color conversion on this image data as appropriate. The display controller 442 includes a display controller 442 that generates image data in a different format, and a rendering processor 443 that develops an image for each frame in a frame memory 444 based on the image data processed by the display controller 442. The image processing section 440 outputs the image data developed in the frame memory 444 to the light modulation device driving section 415.

In the PJ mode described above, the drawing processing unit 443 performs editing to add or delete a graphic image to the image developed in the frame memory 444 according to the coordinates of the position indicated by the input means 402, and generates or updates the image. conduct. As a result, a figure corresponding to the operation of the input means 402 is drawn and projected onto the screen SC.

On the other hand, the input means 402 includes, in addition to the above-mentioned operation switch 403, an attitude switch 404 whose output changes depending on the attitude of the input means 402. The posture switch 404 is a switch that is turned off when the input means 402 is horizontal, and turned on when the input means 402 is tilted, for example. Further, the posture switch 404 may be a switch whose resistance value changes depending on the magnitude of the inclination of the input means 402, for example.

The input means 402 includes a transmitter 474 that outputs an infrared signal. The transmitter 474 includes, for example, a light emitting unit such as an infrared LED, a light emission control circuit, and a power source. The transmitter 474 periodically transmits an infrared signal while the input means 402 is powered on. The transmitter 474 modulates the infrared signal according to the operation state (on/off) of the operation switch 403 and the output state of the attitude switch 404 and transmits the modulated infrared signal. The transmitter 474 transmits data indicating whether the operation switch 403 is on/off and the attitude switch 404 is on/off, for example, using a method compliant with the IrDA standard.

The projector 400 includes a position instruction operation detection section 450 that detects an operation by the input means 402. The position pointing operation detection unit 450 is configured to include an imaging unit 451, a reception unit 452, an imaging control unit 453, a pointer detection unit 454, and a coordinate calculation unit 455. The position pointing operation detecting section 450 functions as an operation detecting means and an attitude detecting means.

The imaging unit 451 is arranged to face the same direction as the projection unit 423, and includes an imaging optical system having an angle of view that covers the range in which the projection unit 423 projects an image onto the screen SC, an imaging element made of a CCD or CMOS, and , has an interface circuit, etc. that reads out and outputs the detection value of the image sensor. Further, the imaging control unit 453 causes the imaging unit 451 to perform imaging to generate captured image data. When the imaging unit 451 executes imaging under the control of the imaging control unit 453, the image projected on the screen SC and the input means 402 operated on or near the screen SC are imaged.

Further, the receiving unit 452 receives and decodes the infrared signal transmitted by the transmitting unit 474, and outputs received data. The received data output by the receiving unit 452 includes data indicating the operating states of the operation switch 403 and the posture switch 404.

The indicator detection unit 454 detects the position of the input means 402 based on the captured image data generated by the imaging control unit 453. The indicator detection unit 454 detects a shape similar to the input means 402 from the captured image data, executes a process of cutting out an image of the input means 402, and specifies the direction in which the tip of the input means 402 is facing. In addition, the indicator detection unit 454 detects the operation state of the operation switch 403 and the attitude of the input means 402 based on the received data output by the reception unit 452. Thereby, the indicator detection unit 454 identifies the position of the input means 402 in the captured image data and the orientation of the input means 402.

The coordinate calculation unit 455 calculates the coordinates of the position that the tip of the input unit 402 is pointing to on the screen SC based on the position of the input unit 402 in the captured image data specified by the pointer detection unit 454 and the orientation of the input unit 402, That is, the coordinates of the indicated position are calculated. The coordinates calculated by the coordinate calculation unit 455 here are, for example, coordinates based on the area (projection area) on which the projection unit 420 projects the projection image onto the screen SC. The coordinate calculation unit 455 outputs data indicating the calculated coordinates of the indicated position, the orientation of the input means 402 specified by the pointer detection unit 454, the operation state of the operation switch 403, etc. to the control unit 430. Further, the coordinate calculation unit 455 may output captured image data captured by the imaging unit 451 to the control unit 430.

The control unit 430 realizes the functions of the projection control unit 431, the detection control unit 432, the function control unit 433, and the output control unit 434 by reading and executing the control program 436 stored in the storage unit 435, Controls each part of the projector 400.

The projection control unit 431 detects the content of the operation performed by the operator based on the operation data input from the operation detection unit 417, and controls the image processing unit 440, light modulation device drive unit 415, and projection according to this operation. The optical system drive section 416 and the light source drive section 414 are controlled to project an image onto the screen SC. The projection control unit 431 also controls the projection optical system drive unit 416 to cause the projection unit 423 to perform focus adjustment, zoom adjustment, aperture adjustment, and the like.

The detection control unit 432 controls the position instruction operation detection unit 450 to detect the indicated position of the input means 402 and the attitude of the input means 402, and detects the detected indicated position and the attitude of the input means 402. get. Furthermore, the detection control unit 432 controls the position indication operation detection unit 450 to detect the operation state of the operation switch 403 included in the input means 402. Thereby, the detection control unit 432 detects that the user has performed an operation using the input means 402 and the state of the input means 402 at the time of the operation.

The detection control unit 432 distinguishes the operation of the input means 402 into a normal operation and a special operation based on the attitude of the input means 402. The normal operation is an operation of pressing the tip of the input means 402 against the screen SC. When a normal operation is performed, the image of the input means 402 extracted from the captured image data by the indicator detection unit 454 faces the screen SC, and the operation switch 403 is turned on. When the user performs drawing or instructions on the toolbar 411 (FIG. 16), normal operations are performed using the input means 402.

The special operation is an operation that causes the input means 402 to take a specific posture. This specific posture is, for example, an operation that makes drawing impossible, and specifically, an operation that makes the input means 402 parallel to the screen SC, and an operation that makes the input means 402 parallel to the screen SC. The input means 402 may be moved vertically or horizontally while performing these operations. Another specific example is an operation in which the distal end of the input means 402, that is, the end opposite to the operation switch 403, is directed toward the screen SC. Since the operation switch 403 cannot be pressed against the screen SC while these operations are being performed, these operations correspond to "operations that do not allow drawing." The state of the input means 402 to be detected as a special operation and the conditions for its detection are set in advance and stored in the storage unit 435.

Among these operations, for example, the operation of making the input means 402 parallel to the screen SC and the operation of making the input means 402 parallel and horizontal to the screen SC are normally performed based on the state of the attitude switch 404. Can be detected separately from manipulation. Further, an operation to direct the end portion of the input means 402 toward the screen SC can be detected as being distinguished from a normal operation based on the orientation of the image of the input means 402 extracted from the captured image data of the imaging section 451. Furthermore, the operation of moving the input means 402 vertically or horizontally with the input means 402 parallel to the screen SC, or with the input means 402 parallel to the screen SC and horizontal, depends on the detection state of the posture switch 404, By comparing the captured image data of the imaging unit 451 with the image of the input means 402 over time, it is possible to detect the operation separately from the normal operation. Further, since the special operation is an operation that does not allow drawing, the condition for detecting the special operation may include that the operation switch 403 is not turned on. Specific conditions for detecting special operations in this way are stored in the storage unit 435.

The detection control unit 432 determines the conditions stored in the storage unit 435, such as the image of the input means 402 detected by the indicator detection unit 454 from the captured image data of the imaging unit 451, the detection states of the operation switch 403 and the attitude switch 404, etc. Detect special operations if the conditions are met.

The function control unit 433 controls execution of the functions of the projector 400 in accordance with the operation by the input means 402 detected by the detection control unit 432 and the operation mode of the projector 400 at the time of the operation.

As described above, the display device 2 switches between the PJ mode in which the projector 400 performs drawing and generates an image based on the operation of the input means 402, and the PC mode in which the external device 6 performs drawing and generates an image. and execute it. Furthermore, in the PJ mode, the projector 400 switches and executes a drawing mode in which drawing is performed and a drawing setting mode in which various settings related to the operation of the drawing mode are made. On the other hand, in the PC mode, the external device 6 switches and executes a drawing mode for drawing, a drawing setting mode for making various settings related to the operation of the drawing mode, and the above-mentioned PC operation mode.

For example, when the display device 2 is operating in the PJ mode and the projector 400 is executing the drawing mode, when the detection control unit 432 detects a normal operation, the function control unit 433 changes the position indicated by the input unit 402. The coordinates of are output to the image input section 441, and the drawing processing section 443 executes drawing. Further, when the detection control unit 432 detects a normal operation while the projector 400 is executing the drawing setting mode, the function control unit 433 outputs the coordinates of the position indicated by the input means 402 and the settings to be projected on the screen SC. The content specified by this operation is specified based on the position of the icon, etc. on the screen, and settings, etc. are performed.

When the detection control unit 432 detects a special operation when the display device 2 is operating in the PJ mode, the function control unit 433 executes a function different from the normal operation. Functions to be executed in response to special operations are assigned in association with the drawing mode and each operation mode other than the drawing mode. These functions are functions other than drawing. When the special operation is detected, the function control unit 433 executes the assigned function corresponding to the operating mode being executed.

Further, when the display device 2 is operating in the PC mode, when the detection control unit 432 detects a normal operation and a special operation, the function control unit 433 outputs an instruction from the input means 402 detected by the position instruction operation detection unit 450. The output control unit 434 causes the external device 6 to output data indicating the position and data indicating whether the detected operation is a special operation or a normal operation.

The external device 6 includes a control section 501 that centrally controls each section of the external device 6. The control unit 501 is realized by a CPU, a ROM that stores basic control programs executed by the CPU, and a RAM that temporarily stores programs executed by the CPU and data processed by the CPU. The external device 6 also includes an input unit 506 that detects input operations using input devices including a keyboard and a pointing device such as a mouse, and a monitor 508 that monitors the contents of the input operations detected by the input unit 506 and the processing results of the control unit 501. an external interface 509 connected to the I/F unit 405 of the projector 400 via a cable 401a, various programs including the control program 111 executed by the control unit 501, and these programs. The storage unit 510 stores data processed by the computer.

The control unit 501 implements the functions of the device control unit 502, image selection unit 503, image output unit 504, and drawing processing unit 505 by executing the control program 511.

The device control unit 502 executes processing for controlling the projector 400 in the PC mode, processing for receiving data transmitted from the projector 400, and the like.

When outputting image data to the projector 400, the image selection unit 503 selects image data to be output according to the operation detected by the input unit 506. For example, the image selection unit 503 may select image data directly specified by a user operation from among the image data (not shown) stored in the storage unit 510, or may It is also possible to select image data within.

The image output unit 504 outputs the image data selected by the image selection unit 503 to the projector 400. Further, the image output unit 504 may perform a process of converting the resolution and frame rate of the selected image data in accordance with the specifications of the projector 400.

In addition, in the PC mode, the device control unit 502 collects data indicating the indicated position of the input means 402 detected by the position instruction operation detection unit 450 of the projector 400, data regarding the attitude of the input means 402, and data indicating that the detected operation is special. Data indicating whether the operation is an operation or a normal operation is received from the projector 400. The device control unit 502 executes functions assigned in advance to each of the operation modes, normal operation, and special operation of the external device 6. For example, in a drawing mode in which the external device 6 performs drawing, the device control unit 502 outputs coordinate data of the designated position to the drawing processing unit 505 for normal operation, and causes the drawing processing unit 505 to execute drawing. Furthermore, when the projector 400 detects a special operation in the drawing mode, it executes switching to an operation mode other than the drawing mode.

In the PC mode, when the coordinates of the position indicated by the input means 402 are input from the device control unit 502, the drawing processing unit 505 executes a drawing process to generate or update an image according to the coordinates, and outputs image data. do. The image data output by the drawing processing unit 505 is output to the projector 400 by the image output unit 504.

FIG. 19 is an example of a functional block diagram showing the functions of the display device 2 related to determining the storage range based on how handwritten data overlaps with the area in this embodiment. The functions of the display device 2 explained in FIG. 9 can also be used in this embodiment as shown in FIG.

That is, the I/F unit 405 corresponds to the image acquisition unit 31. The control section 430 includes the video reception section 11, screen composition section 13, and storage range determination section 14 shown in FIG. Further, the image processing section 440 has a display control section 16. Further, the coordinate calculating section 455 corresponds to the stroke receiving section 12 and the stroke processing section 32.

<Position detection>
The method by which the display device 2 of this embodiment determines the indicated position of the input means 402 will be explained.

(i) Method of determining from the attitude of the input means In addition to the operation switch 403 that is turned on when the user presses the tip of the input means 402 against the screen SC, the output of the input means 402 is switched depending on the attitude of the input means 402. A posture switch 404 is provided. The posture switch 404 is a switch that is turned off when the input means 402 is horizontal, and turned on when the input means 402 is tilted, for example. Further, the posture switch 404 may be a switch whose resistance value changes depending on the magnitude of the inclination of the input means 402, for example.

The input means 402 includes a transmitter 474 that outputs an infrared signal. The transmitter 474 includes, for example, a light emitting unit such as an infrared LED, a light emission control circuit, and a power source. The transmitter 474 periodically transmits an infrared signal while the input means 402 is powered on. The transmitter modulates and transmits the infrared signal depending on the operation state (on/off) of the operation switch 403 and the output state of the attitude switch 404. The transmitter 474 transmits data indicating whether the operation switch 403 is on/off and the attitude switch 404 is on/off, for example, using a method compliant with the IrDA standard.

The projector 400 uses a camera to image the screen SC at an angle of view that covers the range in which the image is projected onto the screen SC. The projector 400 detects the position of the input means 402 based on captured image data captured and generated by the camera. The projector 400 detects a shape similar to the input means 402 from the captured image data, executes processing to cut out the image of the input means 402, and specifies the direction in which the tip of the input means 402 is facing. Furthermore, the projector 400 detects the operation state of the operation switch 403 and the attitude of the input means 402 based on the received data transmitted from the input means 402. Thereby, the projector 400 specifies the position of the input means 402 in the captured image data and the orientation of the input means 402.

Furthermore, the projector 400 calculates the coordinates of the position pointed to by the tip of the input means 402 on the screen SC, that is, the coordinates of the indicated position, based on the position of the input means 402 in the captured image data and the orientation of the input means 402. . The coordinates calculated by the projector 400 here are, for example, coordinates based on a region (projection region) in which a projection image is projected onto the screen SC.

(ii) Detection of distance by sound waves or radio waves and detection of horizontal position by camera When the user presses the tip of the input device 402 against the screen SC for handwriting, the operation switch 403 is turned on and the light-emitting part that emits light is emitted, for example, from the tip of the input means 402. has. The wavelength of the light is near-infrared or infrared, so it is invisible to the user's eyes. The projector 400 has a camera that captures an image of the light emitting section, analyzes the image, and specifies the position of the input means 402 in a direction parallel to the screen SC. Further, the input means 402 emits sound waves along with light emission, and the projector 400 calculates the distance based on the arrival time of the sound waves. The position of the input means 402 can be specified based on the direction and distance. Radio waves may be used instead of sound waves.

(iii) Detecting the position of the input means using an infrared curtain In this case, the input means 402 does not need to have a light emitting section or a transmitting section. For example, it can be handwritten with the user's finger. The projector 400 irradiates infrared light so as to cover the screen SC. An invisible infrared curtain is placed over the screen SC. When the input means 402 touches the curtain, the input means 402 reflects infrared rays, and the reflected light is imaged with a camera. Thereby, the position of the input means 402 in the direction parallel to the screen SC is specified. Further, the distance is calculated based on the time from when the infrared rays are irradiated to when the infrared rays are reflected. The position of the input means 402 can be specified based on the direction and distance.

<Display example of video with split screen>
FIG. 20 shows a display example of a screen when videos input from a plurality of external devices 6 are displayed in this embodiment. As shown in FIG. 20, since a plurality of external devices 6a are connected, the screen is displayed divided into a first area 101 and a second area 102.

<Summary>
According to this embodiment, in addition to the effects of Embodiment 1, also in the projector-type display device 2, the display device to which a plurality of external devices are connected divides the screen and simultaneously displays images of the plurality of external devices. When saving video and handwritten data in a state, the storage range can be determined based on how the handwritten data overlaps with the area.

Note that the display device 2 using a projector projection method described in this embodiment is an example, and the method for displaying cut characters of video input described in the first embodiment can be widely applied to the display device 2 using a projector.

<Other application examples>
Although the best mode for carrying out the present invention has been described above using examples, the present invention is not limited to these examples in any way, and various modifications can be made without departing from the gist of the present invention. and substitutions can be added.

Further, the configuration examples shown in FIGS. 6 and 9 are divided according to main functions in order to facilitate understanding of the processing of the display device 2. The present invention is not limited by the method of dividing the processing units or the names thereof. Furthermore, the processing of the display device 2 can be divided into more processing units depending on the processing content. Furthermore, one processing unit can be divided to include more processing.

Each function of the embodiments described above can be realized by one or more processing circuits. Here, the term "processing circuit" as used herein refers to a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, or a processor designed to execute each function explained above. This includes devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules.

Note that the display device 2 is not limited to the above. The display device 2 is, for example, an output device such as a PJ (Projector), a digital signage, a HUD (Head Up Display) device, an industrial machine, an imaging device, a sound collection device, a medical device, a network home appliance, or a connected car. , a car navigation system, a notebook PC, a mobile phone, a smartphone, a tablet terminal, a game console, a PDA, a digital camera, a wearable PC, a desktop PC, or the like.

1 Image processing system 2 Display device 3 Display 4 Electronic pen

Japanese Patent Application Publication No. 2013-114412 JP2013-222280A

Claims (12)

A display device that displays images input from multiple external devices in different areas of the screen,
a contact detection unit that receives data input based on the contact position of the input means with respect to the screen;
When saving images input from a plurality of external devices, a storage range in which the area where the data exists is determined as the storage range based on the position of the data received by the contact detection unit and the position of the area. A decision section,
a data storage unit that acquires and stores image data in the storage range determined by the storage range determination unit from the screen ;
The storage range determining unit includes:
If the data exists only in one area, only the area where the data exists is determined as the storage range,
If the data exists in multiple areas, each area where the data exists is determined as the storage range,
If the data exists outside the area, determining the area where the data exists and a circumscribing rectangle of the data as the storage range;
A display device characterized in that, when the data does not exist on the screen, all of a plurality of areas are determined as the storage range . When saving images input from multiple external devices,
The data storage unit stores the image data of the video of the region for each region, or if the data exists in the region, stores the video of the region and the image data of the data,
Storing the image data of images of multiple areas, or, if the data exists in multiple areas, storing image data including images of multiple areas and the data;
If the data exists outside the area, save the image data of a circumscribed rectangle between the data and one or more areas in which the data exists;
Save the image data of the video of all multiple areas,
If the data exists only in one area, displaying the image of the area and the image data of the data;
If the data exists in multiple areas, displaying image data including images of the multiple areas and the data;
If the data is outside the area, displaying the image data of a circumscribed rectangle between the data and one or more areas where the data exists;
2. The display device according to claim 1 , wherein when the data does not exist on the screen, image data of videos of all the plurality of regions is displayed. The display device according to claim 2 , wherein when the data exists in a plurality of areas and the data spans the plurality of areas, the display device displays images of the plurality of areas and image data including the data. . Record the time when the data was input,
If the data exists in multiple areas and the difference in the smallest time between the areas among the times when the data was input to each area is less than a threshold, display images of the multiple areas and image data including the data. The display device according to claim 2, characterized in that: Record the time when the data was input,
If the data exists in multiple areas and the data does not span multiple areas,
4. The display device according to claim 3 , wherein the display device displays an image of the area in which the data whose input time is the latest exists and the image data of the data. Record the time when the data was input,
If the data exists in a plurality of regions, and the difference in the smallest time between the regions among the times at which the data was input to each region is not less than a threshold, the region in which the data exists at the latest input time; 5. The display device according to claim 4, wherein the display device displays the video of the image and the image data of the data. Recording the time when the external device was connected to the display device,
If the data exists in multiple areas and the data does not span multiple areas,
4. The display device according to claim 3, wherein the display device displays the video of the area of the last connected external device and the image data of the data. Record the time when the data was input,
Recording the time when the external device was connected to the display device,
If the data exists in multiple areas, and the difference in the smallest time between the areas among the times when the data was input to each area is not less than a threshold, the image of the area of the external device connected last and The display device according to claim 4, wherein the display device displays the image data of the data. If the data exists in multiple areas and the data does not span multiple areas,
4. The display device according to claim 3 , wherein the image data of the area and the image data of the area where the image does not change are displayed. Record the time when the data was input,
If the data exists in a plurality of areas, and the difference in the smallest time between the areas among the times at which the data was input to each area is not less than a threshold, the image of the area where the image has not changed and the data 5. The display device according to claim 4 , wherein the display device displays the image data of. A display method performed by a display device that displays images input from multiple external devices in different areas of the screen, the display method comprising:
a step in which the contact detection unit receives data input based on a contact position of the input means with respect to the screen;
When saving images input from a plurality of external devices, a storage range determining unit saves the area where the data exists based on the position of the data received by the contact detection unit and the position of the area. a step of determining a range;
a step in which a data storage unit acquires and stores image data in the storage range determined by the storage range determination unit from the screen ;
The storage range determining unit includes:
If the data exists only in one area, only the area where the data exists is determined as the storage range,
If the data exists in multiple areas, each area where the data exists is determined as the storage range,
If the data exists outside the area, determining the area where the data exists and a circumscribing rectangle of the data as the storage range;
A display method characterized in that, when the data does not exist on the screen, all of a plurality of areas are determined as the storage range . A display device that displays images input from multiple external devices in different areas of the screen.
a contact detection unit that receives data input based on the contact position of the input means with respect to the screen;
When saving images input from a plurality of external devices, a storage range in which the area where the data exists is determined as the storage range based on the position of the data received by the contact detection unit and the position of the area. A decision section,
functioning as a data storage unit that acquires and stores image data in the storage range determined by the storage range determination unit from the screen;
The storage range determining unit includes:
If the data exists only in one area, only the area where the data exists is determined as the storage range,
If the data exists in multiple areas, each area where the data exists is determined as the storage range,
If the data exists outside the area, determining the area where the data exists and a circumscribing rectangle of the data as the storage range;
A program that determines all of a plurality of areas as the storage range when the data does not exist on the screen .

Images