JP2021051256A - Display unit, display method, and program - Google Patents

Abstract

To provide a display unit that can make clearer the boundary between an image and a display area.SOLUTION: A display unit 2 displays an acquired image, and comprises: a color determination unit 53 that, when the aspect ratio of the image and the aspect ratio of a display area where the image is displayed are different, determines the color of a margin generated when the image is displayed in the display area according to the colors used in the image; and an image creation unit 34 that displays the image in the display area and displays the margin with the color determined by the color determination unit.SELECTED DRAWING: Figure 3

Description

The present invention relates to display devices, display methods, and programs.

A display device that displays information on a display such as a liquid crystal or an organic EL or a projector or the like is known. For example, there are a display device that displays data handwritten by a user with a pen or a finger on a touch panel, and a display device that displays an advertisement or the like on a passerby.

The aspect ratio of the display or the display area of these display devices varies depending on the display device, and a technique for displaying an image according to the aspect ratio of the display has been devised (see, for example, Patent Document 1). Patent Document 1 discloses a technique in which an image output device side generates an image according to a display area.

However, in the conventional technique, there is a problem that the boundary between the image and the display area becomes unclear when the display area of the image and the aspect ratio of the image are different. For example, when the display area of the image and the aspect ratio of the image are different, the display device may fill the margins of the display area generated on the left, right, or top and bottom of the image with a uniform color for display. However, when the color of the image and the color of the margin outside the image are the same, the boundary between the image and the margin is difficult to understand, which may make it difficult for the user to operate or cause a sense of discomfort.

In view of the above problems, an object of the present invention is to provide a display device capable of clarifying the boundary between an image and a display area.

In view of the above problems, the present invention is a display device for displaying an acquired image, and when the aspect ratio of the image and the aspect ratio of the display area in which the image is displayed are different, the color used in the image is used. A color determination unit that determines the color of the margin generated when the image is displayed in the display area, and a color determination unit that displays the image in the display area and displays the margin in the color determined by the color determination unit. It is characterized by having an image generation unit to be used.

It is possible to provide a display device that can make the boundary between an image and a display area clearer.

It is a figure explaining the comparison technique when the display device displays an image. It is a figure explaining the inconvenience when the boundary between an image and a margin is difficult to understand. It is a figure which shows the display example of the margin when the display apparatus acquired the image of the aspect ratio different from the display. This is an example of an overall configuration diagram of the image processing system of the present embodiment. This is an example of a hardware configuration diagram of a display device. This is an example of a functional block diagram of a display device. This is an example of a functional block diagram of the file processing unit. It is a conceptual diagram which shows page data. It is a conceptual diagram which shows stroke data. It is a conceptual diagram which shows the coordinate array data. It is a conceptual diagram which shows the media data. This is an example of a configuration diagram of each image layer. This is an example of a functional block diagram showing the function of the display device regarding the determination of the margin color according to the aspect ratio of the image read by the display device and the display. It is an example of the figure explaining the display direction. It is an example of a figure explaining an image and a color included. This is an example of a diagram showing representative colors in a palette. It is an example of the figure explaining the color which is not used at the edge of an image. It is an example of the figure explaining the scaling of an image. It is an example of the figure explaining the method of determining the position of a margin. This is an example of a flowchart showing a procedure for determining the margin color of the display area according to the aspect ratio of the image read by the display device and the display. This is an example of a diagram for explaining a method of determining a color that is not commonly used for each page among the representative colors. This is an example of a flowchart showing a procedure for determining the margin color of the display area according to the aspect ratio of the image read by the display device and the display in the case of a plurality of pages. This is an example of a diagram for explaining the margin reduced by the image generation unit according to the coordinates of the stroke. This is an example of a flowchart showing a procedure for changing the size of the margin and capturing an image. It is a figure which shows the other configuration example of a display device. It is a figure which shows the other configuration example of a display device. It is a figure which shows the other configuration example of a display device. It is a figure which shows the other configuration example of a display device.

Hereinafter, as an example of the embodiment for carrying out the present invention, the display device and the display method performed by the display device will be described with reference to the drawings.

<Comparison technology>
In explaining the display device of the present embodiment, a suitable comparison technique will be described.
FIG. 1 is a diagram illustrating a comparison technique when the display device 2 displays the image 11. First, the display device 2 includes a display device 2 having a vertically long display 3 (hereinafter referred to as a vertical display device 2) and a display device 2 having a horizontally long display 3 (hereinafter referred to as a horizontal display device 2).

In FIGS. 1A to 1C, the horizontal display device 2 displays the vertically long image 11. The image 11 shown in FIG. 11A is stored in a file acquired by the display device 2, such as a PDF file. In FIG. 1, the entire display 3 is a display area. As shown in FIG. 1B, the aspect ratios of the image 11 and the display 3 are different. In this case, the horizontal display device 2 scales and displays the image 11 so that the longer side of the image 11 matches the vertical length of the display 3.

In FIG. 1C, the image 11 is displayed substantially in the center of the display 3 in the left-right direction. However, since the background of the image 11 and the margin 13 of the display (the rest of the display area generated when the image 11 is displayed in the display area) are both all white, the boundary between the image and the margin 13 becomes difficult to understand. There is.

In FIGS. 1 (d) to 1 (f), the vertical display device 2 displays the horizontally long image 11. Similarly, the vertical display device 2 scales and displays the image 11 so that the longer side of the image 11 matches the lateral length of the display 3.

In FIG. 1 (f), the image 11 is displayed substantially in the center of the display 3 in the vertical direction. However, since the background of the image 11 and the margin 13 of the display are both all white, it is difficult to understand the boundary between the image and the margin 13.

FIG. 2 is a diagram illustrating inconvenience when the boundary between the image 11 and the margin 13 is difficult to understand. First, as shown in FIG. 2A, since it is difficult for the user to understand what kind of shape (portrait or landscape) the image 11 displayed on the display device 2 was, the displayed image 11 is displayed. It may be difficult to determine if the file you are trying to view.

Further, as shown in FIG. 2B, the user can write handwritten data on the image 11 as a function of the display device 2. It is also possible to capture (save or capture) this writing together with the image 11. At the time of capture, the display device 2 may capture the entire display 3 or capture only the range of the image 11. In FIG. 2B, the user writes the handwritten data 8 called “ABC” beyond the range of the image 11, but since the display device 2 captures only the range of the image 11, the handwritten data written in the captured image 12 Some parts of are not saved.

As described above, when the boundary between the image 11 and the margin 13 is difficult to understand, it may be difficult for the user to write in the range of the image 11.

<Outline of operation of the display device of this embodiment>
Therefore, when the aspect ratios of the image 11 acquired and displayed and the display 3 are different, the display device 2 of the present embodiment determines the color of the margin 13 according to the color of the image 11. For example, a color not used in the image 11 is set as the color of the margin 13.

FIG. 3 shows a display example of a margin 13 when an image 11 having an aspect ratio different from that of the display 3 is acquired and displayed by the display device 2. FIG. 3A shows the case of the vertical display device 2, and FIG. 3B shows the case of the horizontal display device 2.

As shown in FIGS. 3A and 3B, since the margin 13 is displayed in a color not used in the image 11, the user can easily determine the boundary between the image 11 and the margin 13 of the display.

As described above, in the display device 2 of the present embodiment, since the color of the margin 13 is determined according to the color used in the image 11, the user can easily determine the boundary between the image 11 and the margin 13 of the display. it can.

<Terminology>
The aspect ratio is the ratio of the vertical and horizontal lengths (number of pixels) of a screen or image. Also called aspect ratio. If the aspect ratios are different, it can be said that they are not similar.

The horizontal orientation means that the aspect ratio is larger than 1, and the vertical orientation means that the aspect ratio is smaller than 1 (it does not matter if the aspect ratio is 1). However, in this embodiment, it is defined as follows. When horizontal W and vertical H, it is written as W: H.
Aspect ratio = W / H.

The display area on which the image is displayed is an area of the display on which the image is displayed. The largest display area is the entire display. The display device becomes a part of the display when the window formed on the display becomes a display area or is divided into rectangular frames according to the purpose.

The display position refers to a position in the display area where the image is displayed when the aspect ratios of the display area and the image 11 are different. For example, when the display 3 is oriented horizontally, there is an arbitrary direction in the left-right direction, and when the display is oriented vertically, there is an option in the vertical direction.

The handwritten data is data in which a coordinate point sequence in which the user continuously moves the input means on the display is displayed as a locus. Further, a series of operations in which the user presses the input means on the display, moves the input means continuously, and then releases the input means from the display is called a stroke, and the data handwritten by the stroke may be called stroke data. The handwritten data has one or more stroke data.

<System overview>
FIG. 4 is an overall configuration diagram of the image processing system of the present embodiment. In addition, in FIG. 4, for simplification of the description, only two display devices 2a and 2b and the electronic pens 4a and 4b associated therewith are shown, and three or more display devices and electronic devices are shown. You may use a pen or the like.

As shown in FIG. 4, the image processing system 1 includes a plurality of display devices 2a and 2b, a plurality of electronic pens 4a and 4b, a USB memory 5a and 5b, a notebook PC (Personal Computer) 6a and 6b, and a television (Personal Computer) 6a and 6b. Video) It has conference terminals 7a, 7b, and a PC8. Further, the display devices 2a and 2b and the PC 8 are connected so as to be able to communicate with each other via the communication network 9. Further, the plurality of display devices 2a and 2b are provided with displays 3a and 3b, respectively.

Further, the display device 2a causes the display 3a to display an image drawn by an event (touch of the pen tip of the electronic pen 4a or the pen tail of the electronic pen 4a on the display 3a) caused by the electronic pen 4a. Can be done. In addition to the electronic pen 4a, the image displayed on the display 3a can be changed based on an event (gestures such as enlargement, reduction, page turning, etc.) caused by a hand Ha including the user's finger. it can.

Further, a USB memory 5a can be connected to the display device 2a, the display device 2a can read an electronic file such as a PDF from the USB memory 5a, and the display device 2a can record the electronic file in the USB memory 5a. Further, the display device 2a is connected to the display device 2a via a cable 10a1 capable of communicating according to standards such as DisplayPort, DVI (Digital Visual Interface), HDMI (registered trademark. High-Definition Multimedia Interface) and VGA (Video Graphics Array). PC6a is connected. Then, the display device 2a generates an event by touching the display 3a, and transmits the event information indicating the event to the notebook PC 6a in the same manner as the event from the input device such as a mouse or a keyboard. Similarly, a television (video) conference terminal 7a is connected to the display device 2a via a cable 10a2 capable of communicating according to the above standard. The notebook PC 6a and the video conferencing terminal 7a may communicate with the display device 2a by wireless communication compliant with various wireless communication protocols such as Bluetooth (registered trademark).

On the other hand, at other bases where the display device 2b is installed, similarly to the above, the display device 2b equipped with the display 3b, the electronic pen 4b, the USB memory 5b, the notebook PC 6b, the video conferencing terminal 7b, the cable 10b1, and the cable 10b2. Is used. Further, the image displayed on the display 3b can be changed based on the event generated by the user's hand Hb or the like.

As a result, the image drawn on the display 3a of the display device 2a at one base is also displayed on the display 3b of the display device 2b at the other base, and conversely, on the display 3b of the display device 2b at the other base. The image drawn in is displayed on the display 3a of the display device 2a at one base. As described above, since the image processing system 1 can perform remote sharing processing for sharing the same image at a remote location, it is very convenient to use it for a conference or the like at a remote location.

In the following, when an arbitrary display device among a plurality of display devices is indicated, it is referred to as "display device 2". When indicating an arbitrary display among a plurality of displays, it is indicated as "display 3". When indicating an arbitrary electronic pen among a plurality of electronic pens, it is indicated as "electronic pen 4". When indicating an arbitrary USB memory among a plurality of USB memories, it is indicated as "USB memory 5". When indicating an arbitrary notebook PC among a plurality of notebook PCs, it is indicated as "notebook PC 6". When indicating an arbitrary video conferencing terminal among a plurality of video conferencing terminals, it is indicated as "video conferencing terminal 7". Further, when indicating an arbitrary hand among the hands of a plurality of users, it is indicated as "hand H". When indicating an arbitrary cable among a plurality of cables, it is indicated as "cable 10".

Further, in the present embodiment, an electronic blackboard will be described as an example of the display device 2, but the present invention is not limited to this, and other examples of the display device 2 include an electronic signage (digital signage), sports, weather forecast, and the like. It may be a telestrator to be used, a remote image (video) diagnostic device, or the like. Further, the notebook PC 6 will be described as an example of the information processing terminal, but the present invention is not limited to this, and other examples of the information processing terminal include a desktop PC, a tablet PC, a PDA, a digital video camera, a digital camera, and a game. It may be a terminal capable of supplying an image frame such as a machine. Further, the communication network includes the Internet, LAN (Local Area Network), mobile phone communication network and the like. Further, in the present embodiment, the USB memory will be described as an example of the recording medium, but the present invention is not limited to this, and various recording media such as an SD card may be used as another example of the recording medium.

<Hardware configuration of display device>
FIG. 5 is an example of a hardware configuration diagram of the display device. In FIG. 5, a display device 2 will be described as a display device 2. As shown in FIG. 5, the display device 2 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, an SSD (Solid State Drive) 204, and a network I / It includes an F205 and an external device connection I / F (Interface) 206.

Of these, the CPU 201 controls the operation of the entire display device 2. The ROM 202 stores a program used to drive the CPU 201 such as the CPU 201 and the IPL (Initial Program Loader). The RAM 203 is used as a work area of the CPU 201. The SSD 204 stores various data such as a program for a display device. The network I / F 205 controls communication with the communication network 9. The external device connection I / F 206 is an interface for connecting various external devices. The external device in this case is, for example, a USB (Universal Serial Bus) memory 5, an external device (microphone 240, speaker 250, camera 260).

Further, the display device 2 includes a capture device 211, a GPU 212, a display 3 controller 213, a contact sensor 214, a sensor controller 215, an electronic pen controller 216, a short-range communication circuit 219, an antenna 219a of the short-range communication circuit 219, and a power switch 222. , Selection switches 223, a camera 224, and a direction detection sensor 225.

Of these, the capture device 211 causes the display 3 of the external PC (Personal Computer) 270 to display the image as a still image or a moving image. The GPU (Graphics Processing Unit) 212 is a semiconductor chip that specializes in graphics. The display 3 controller 213 controls and manages the screen display in order to output the output image from the GPU 212 to the display 3 and the like. The contact sensor 214 detects that the electronic pen 290, the user's hand H, or the like has come into contact with the display 3. The sensor controller 215 controls the processing of the contact sensor 214. The contact sensor 214 inputs and detects the coordinates by the infrared blocking method. In the method of inputting the coordinates and detecting the coordinates, two light receiving and emitting devices installed at both upper ends of the display 3 radiate a plurality of infrared rays in parallel with the display 3 and are provided around the display 3. This is a method of receiving light that is reflected by a reflecting member and returns on the same optical path as the light path emitted by the light receiving element. The contact sensor 214 outputs an infrared ID emitted by two light emitting / receiving devices blocked by the object to the sensor controller 215, and the sensor controller 215 specifies a coordinate position which is a contact position of the object. The electronic pen controller 216 determines whether or not there is a touch of the pen tip or a touch of the pen tail on the display 3 by communicating with the electronic pen 290. The short-range communication circuit 219 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The power switch 222 is a switch for switching ON / OFF of the power of the display device 2. The selection switches 223 are, for example, a group of switches for adjusting the brightness and color tone of the display of the display 3.

The camera 224 is an image pickup device built in, for example, the bezel of the display 3, and images the front of the display device 2. If there is a camera 224, it is not necessary to have an external camera like the camera 260.

The direction detection sensor 225 is a sensor that detects the direction of the display 3. The display 3 has a vertical orientation and a horizontal orientation. In the display device 2 of this embodiment, the user can switch the display 3 in portrait or landscape orientation. Such a function is called a pivot function. The rotation itself depends on the mechanism of the stand that supports the display 3, but the direction detection sensor 225 detects the rotation angle of the display 3. For example, the acceleration sensor detects the direction of gravity to detect whether it is in portrait or landscape orientation. Alternatively, the rotation of the stand portion may be detected. The display device 2 rotates the image 11 so that the top and bottom of the image 11 are aligned with each other regardless of whether the display 3 is oriented horizontally or vertically.

Further, the display device 2 includes a bus line 210. The bus line 210 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 201 shown in FIG.

The contact sensor 214 is not limited to the infrared blocking method, but is a capacitive touch panel that specifies a contact position by detecting a change in capacitance, and a contact position is specified by a voltage change between two opposing resistance films. Various detection means such as a resistance film type touch panel and an electromagnetic induction type touch panel that detects the electromagnetic induction generated when a contact object comes into contact with the display unit and specifies the contact position may be used. Further, the electronic pen controller 216 may determine whether or not there is a touch not only on the pen tip and pen end of the electronic pen 290 but also on the portion gripped by the user of the electronic pen 290 and other electronic pen portions.

<Functional configuration of display device>
Subsequently, the functional configuration of the display device 2 will be described with reference to FIGS. 6 to 12. First, the overall functional configuration of the display device 2 will be described with reference to FIG. FIG. 6 is a functional block diagram for explaining the functions of the display device 2 by dividing them into blocks.

The display device 2 has each functional configuration shown in FIG. 6 according to the hardware configuration and program shown in FIG. The display device 2 can be a "sponsoring device" that first starts the remote sharing process, and can also be a "participating device" that later participates in the remote sharing process that has already been started. Further, the display device 2 is roughly divided into both a client unit 20 and a server unit 90.

The client unit 20 and the server unit 90 are functions realized in one housing of the display device 2. When the display device 2 serves as the host device, the client unit 20 and the server unit 90 are realized in the display device 2. Further, when the display device 2 becomes a participating device, the client unit 20 is realized in the display device 2, but the server unit 90 is not realized. That is, in FIG. 4, when the display device 2a is the host device and the display device 2b is the participating device, the client unit 20 of the display device 2a is via the server unit 90 realized in the same display device 2a. Communicates with the client unit 20 of the display device 2b. On the other hand, the client unit 20 of the display device 2b communicates with the client unit of the other display device 2a via the server unit 90 realized in the other display device 2a.

[Functional configuration of client part]
Subsequently, the functional configuration of the client unit 20 will be described mainly with reference to FIGS. 6 to 12. The client unit 20 includes a video acquisition unit 21, a coordinate detection unit 22, an automatic adjustment unit 23, a contact detection unit 24, an event distribution unit 25, an operation processing unit 26, a gesture processing unit 27, an image superimposing unit 28, and an image processing unit 30. , And a communication control unit 60.

Of these, the video acquisition unit 21 acquires the output image of the video output device connected to the cable 10. When the image acquisition unit 21 receives an image signal from the image output device, the image acquisition unit 21 analyzes the image signal, updates the resolution of the image frame which is the display image of the image output device formed by the image signal, and updates the image frame. Image information such as frequency is derived and output to the image acquisition unit 31.

The coordinate detection unit 22 detects the coordinate position on the display 3 that the user has brought into contact with the input means. The handwritten data 8 is a locus in which the user continuously moves the input means on the display 3. Further, the handwritten data 8 may be referred to as a stroke. The input means may be either the electronic pen 4 or the user's hand H. The coordinate detection unit 22 also detects the touched area.

The automatic adjustment unit 23 is activated when the display device 2 is activated, and adjusts the image processing parameters of the sensor camera in the optical sensor method of the contact sensor 214 so that the contact sensor 214 can output an appropriate value to the coordinate detection unit 22. To do.

The contact detection unit 24 detects an event (such as an operation in which the pen tip of the electronic pen 4 or the pen tail of the electronic pen 4 is pressed (touched) on the display 3) caused by the user.

The event distribution unit 25 distributes the coordinate position of the event detected by the coordinate detection unit 22 and the detection result detected by the contact detection unit 24 into each event of stroke drawing, UI operation, and gesture operation.

Here, in the "stroke drawing", when the stroke image (B) shown in FIG. 12 is displayed on the display 3, the user presses the electronic pen 4 on the display 3, and this pressing is performed. This is an event in which the electronic pen 4 is moved in this state, and finally the electronic pen 4 is released from the display 3. By this stroke drawing, for example, the alphabets "S" and "T" are drawn on the display 3. Note that this "stroke drawing" includes not only drawing an image, but also an event of deleting an already drawn image or editing the drawn image.

The "UI operation" is an event in which the user presses a predetermined position with the electronic pen 4 or the hand H when the UI image (A) shown in FIG. 12 is displayed on the display 3. .. By this UI operation, for example, the color and width of the line drawn by the electronic pen 4 are set.

The "gesture operation" is an event in which the user touches or moves on the display 3 with the hand H when the stroke image (B) shown in FIG. 12 is displayed on the display 3. is there. By this gesture operation, for example, by moving the hand H while the user touches the hand H on the display 3, the image can be enlarged (or reduced), the display area can be changed, the page can be switched, and the like. it can.

The operation processing unit 26 executes various operations according to the elements of the UI in which the event has occurred from those determined to be UI operations by the event distribution unit 25. Elements of this UI include, for example, buttons, lists, checkboxes, and text boxes.
The gesture processing unit 27 executes an operation corresponding to the gesture operation determined by the event distribution unit 25.

The image superimposing unit 28 displays the image superimposed by the display superimposing unit 36, which will be described later, as an image on the image output device (display 3 or the like). Further, the video superimposing unit 28 pictures-in-pictures the video sent from another video output device (video conference terminal 7 or the like) with respect to the video from the video output device (notebook PC 6 or the like). Further, the image superimposing unit 28 switches to display the image that has been picture-in-pictured and displayed on a part of the display 3 on the entire display 3.

The image processing unit 30 performs superimposition processing of each image layer as shown in FIG. The image processing unit 30 includes an image acquisition unit 31, a stroke processing unit 32, a UI image generation unit 33, an image generation unit 34, a layout management unit 35, a display superimposition unit 36, a page processing unit 37, a file processing unit 40, and page data. It has a storage unit 300 and a remote license management table 310.

Of these, the image acquisition unit 31 acquires each frame as an image from the image acquired by the image acquisition unit 21. The image acquisition unit 31 outputs the data of this image to the page processing unit 37. This image corresponds to the output image (C) from the video output device (notebook PC 6 or the like) shown in FIG.

The stroke processing unit 32 draws an image, deletes the drawn image, and edits the drawn image based on the event related to the stroke drawing assigned by the event distribution unit 25. The image obtained by drawing the stroke corresponds to the stroke image (B) shown in FIG. Further, each result of drawing, deleting, and editing the image based on the stroke drawing is stored in the operation data storage unit 840 as the operation data described later.

The UI image generation unit 33 generates a UI (user interface) image preset in the display device 2. This UI image corresponds to the UI image (A) shown in FIG.

The image generation unit 34 receives the media data of the page data read from the page data storage unit 300 by the page processing unit 37 from the page processing unit 37. The image generation unit 34 outputs the received media data to the display superimposition unit 36. Further, the image based on this media data corresponds to the background image (D) shown in FIG. The pattern of the background image (D) is a solid color, a grid display, or the like.

The layout management unit 35 provides the display superimposition unit 36 with layout information indicating the layout of each image output from the image acquisition unit 31, the stroke processing unit 32, and the UI image generation unit 33 (or the image generation unit 34). I manage it. As a result, the layout management unit 35 causes the display superimposition unit 36 to display the output image (C) and the stroke image (B) at which position in the UI image (A) and the background image (D). You can instruct whether to hide it.

Based on the layout information output from the layout management unit 35, the display superimposition unit 36 arranges the layout of each image output from the image acquisition unit 31, the stroke processing unit 32, and the UI image generation unit 33 (image generation unit 34). Do.

The page processing unit 37 collects the data of the stroke image (B) and the data of the output image (C) into one page data and stores it in the page data storage unit 300. The data of the stroke image (B) forms a part of the page data as the stroke array data (each stroke data) indicated by the stroke array data ID shown in FIG. The data of the output image (C) forms a part of the page data as the media data indicated by the media data ID shown in FIG. Then, when this media data is read from the page data storage unit 300, it is treated as the data of the background image (D).

Further, the page processing unit 37 transmits the media data of the page data once stored to the display superimposition unit 36 via the image generation unit 34, so that the image superimposition unit 28 displays the background image (D). It can be redisplayed in 3. Further, the page processing unit 37 can return the stroke array data (each stroke data) of the page data to the stroke processing unit 32 so that the stroke can be re-edited. Further, the page processing unit 37 can also delete or duplicate the page data.

That is, the data of the output image (C) displayed on the display 3 at the time when the page processing unit 37 stores the page data in the page data storage unit 300 is temporarily stored in the page data storage unit 300, and then stored in the page data storage unit 300. When it is read from the page data storage unit 300, it is read as media data indicating the background image (D). Then, the page processing unit 37 outputs the stroke array data indicating the stroke image (B) among the page data read from the page data storage unit 300 to the stroke processing unit 32. Further, the page processing unit 37 outputs the media data indicating the background image (D) among the page data read from the page data storage unit 300 to the image generation unit 34.

The display superimposition unit 36 is an output image (C) from the image acquisition unit 31, a stroke image (B) from the stroke processing unit 32, a UI image (A) from the UI image generation unit 33, and an image generation unit 34. The background image (D) of the above is superimposed according to the layout specified by the layout management unit 35. As a result, as shown in FIG. 12, each of the UI image (A), the stroke image (B), the output image (C), and the background image (D) is in the order in which the user can see the images even if they overlap each other. It is composed of Leia.

Further, the display superimposition unit 36 can switch between the image (C) and the image (D) shown in FIG. 12 and superimpose the image (A) and the image (B) exclusively. .. For example, when the image (A), the image (B), and the image (C) are initially displayed and the cable 10 between the display device 2 and the video output device (notebook PC 6 or the like) is disconnected, By the designation of the layout management unit 35, the image (C) can be excluded from the superimposition target and the image (D) can be displayed. In this case, the display superimposing unit 36 also performs display enlargement, display reduction, and display area movement processing.

The page data storage unit 300 stores page data as shown in FIG. FIG. 8 is a conceptual diagram showing page data. The page data is data for one page (stroke array data (each stroke data) and media data) displayed on the display 3. Since there are many types of parameters included in the page data, the contents of the page data will be described here by dividing them into FIGS. 8 to 11.

As shown in FIG. 8, the page data includes a page data ID for identifying an arbitrary page, a start time indicating the time when the display of this page is started, and rewriting the contents of the page by strokes, gestures, or the like. The end time indicating the time when is no longer performed, the stroke array data ID for identifying the stroke array data generated by the stroke by the electronic pen 4 or the user's hand H, and the media data ID for identifying the media data are associated with each other. Is remembered. The stroke array data is data for displaying the stroke image (B) shown in FIG. 12, which will be described later, on the display 3. The media data is data for displaying the background image (D) shown in 16 below on the display 3.

With such page data, for example, when the user draws the alphabet "S" with the electronic pen 4, it is written in one stroke, so that the alphabet [S] of one character is indicated by one stroke data ID. However, when the user draws the alphabet "T" with the electronic pen 4, since it is written in two strokes, the alphabet "T" of one character is indicated by two stroke data IDs.

Further, the stroke array data shows detailed information as shown in FIG. FIG. 9 is a conceptual diagram showing stroke array data. As shown in FIG. 9, one stroke array data is represented by a plurality of stroke data. Then, one stroke data includes a stroke data ID for identifying the stroke data, a start time indicating the writing start time of one stroke, an end time indicating the writing end time of one stroke, and a stroke color. The width of the stroke and the coordinate array data ID for identifying the array of passing points of the stroke are shown.

Further, the coordinate array data shows detailed information as shown in FIG. FIG. 10 is a conceptual diagram showing coordinate array data. As shown in FIG. 10, the coordinate array data is one point (X coordinate value, Y coordinate value) on the display 3, and the time (ms) of the difference from the start time of the stroke when passing through this one point. ), And each information of the pen pressure of the electronic pen 4 at this one point is shown. That is, a set of one point shown in FIG. 10 is shown by one coordinate array data shown in FIG. For example, when the user draws the alphabet "S" with the electronic pen 4, it is written in one stroke, but since the user passes through a plurality of passing points by the time the drawing of the "S" is completed, the coordinate array data has these multiple passing points. Information is shown.

Further, the media data among the page data shown in FIG. 8 shows detailed information as shown in FIG. FIG. 11 is a conceptual diagram showing media data. As shown in FIG. 11, as for the media data, the media data ID in the page data shown in FIG. 8, the data type of the media data, and the page data are stored in the page data storage unit 300 from the page processing unit 37. The recorded time, the position (X coordinate value, Y coordinate value) of the image displayed on the display 3 by the page data, the size (width, height) of the image, and the data indicating the contents of the media data are shown in association with each other. Has been done. Of these, the position of the image displayed on the display 3 by the page data is displayed by the page data when the coordinates of the upper left corner of the display 3 are (X coordinate value, Y coordinate value) = (0,0). It shows the position of the upper left corner of the image to be displayed.

Further, returning to FIG. 6, the remote license management table 310 manages the license data necessary for executing the remote sharing process. In the remote license management table 310, the product ID of the display device 2, the license ID used for authentication, and the expiration date of the license are managed in association with each other.

<Functional configuration of file processing unit>
Subsequently, the functional configuration of the file processing unit 40 shown in FIG. 6 will be described with reference to FIG. 7. Note that FIG. 7 is a functional block diagram of the file processing unit.

The file processing unit 40 includes a recovery processing unit 41, a file input unit 42a, a file output unit 42b, a file conversion unit 43, a file transmission unit 44, an address book input unit 45, a backup processing unit 46, a backup output unit 47, and a setting management unit. It has 48, a setting file input unit 49a, and a setting file output unit 49b. Further, the file processing unit 40 has an address book management table 410, a backup data storage unit 420, a setting file storage unit 421, and a connection destination management table 440.

Of these, the recovery processing unit 41 detects the abnormal termination after the display device 2 abnormally terminates, and recovers the unsaved page data. For example, in the case of normal termination, the page data is recorded as a PDF file in the USB 5 via the file processing unit 40, but in the case of abnormal termination such as when the power is turned off, the page data is recorded in the page data storage unit 300. It remains to be done. Therefore, when the power is turned on again, the recovery processing unit 41 recovers by reading the page data from the page data storage unit 300.

The file input unit 42a reads the PDF file from the USB memory 5 and stores each page as page data in the page data storage unit 300. The PDF file is the media data of the page data, that is, the background image. This PDF file may be captured by the display device 2 and stored in the USB memory 5.

The file conversion unit 43 converts the page data stored in the page data storage unit 300 into a PDF format file. The file output unit 42b records the PDF file output by the file conversion unit 43 in the USB memory 5. The file transmission unit 44 attaches the PDF file generated by the file conversion unit 43 to an e-mail and transmits the PDF file. The destination of this file is that the display superimposition unit 36 displays the contents of the address book management table 410 on the display 3, and the file transmission unit 44 accepts the selection of the destination by the operation of an input device such as a touch panel by the user. It is determined.

The address book input unit 45 reads the e-mail address list file from the USB memory 5 and manages it in the address book management table 410.

The backup processing unit 46 backs up the file output by the file output unit 42b and the file transmitted by the file transmission unit 44 by storing it in the backup data storage unit 420. If the user does not set the backup, the backup process will not be performed.

The backup output unit 47 stores the backed up file in the USB memory 5. At the time of this storage, a password is input for security by the operation of an input device such as a touch panel by the user.

The setting management unit 48 manages by storing and reading various setting information of the display device 2 in the setting file storage unit 421. Examples of the various setting information include network settings, date and time settings, region and language settings, mail server settings, address book settings, connection destination list settings, and backup settings. The network settings include, for example, the IP address setting of the display device 2, the netmask setting, the default gateway setting, the DNS (Domain Name System) setting, and the like.

The setting file output unit 49b records various setting information of the display device 2 in the USB memory 5 as a setting file. Due to the security of the configuration file, the user cannot see the contents.

The setting file input unit 49a reads the setting file stored in the USB memory 5 and reflects various setting information in various settings of the display device.

The connection destination input unit 50 reads a list file of connection destination IP addresses for remote sharing processing from the USB memory 5 and manages them in the connection destination management table 440. When the display device 2 is a participating device that intends to participate in the remote sharing process, the connection destination management table 440 reduces the time and effort for the user of the participating device to input the IP address of the display device that serves as the host device. Therefore, it is a table to be managed in advance. In the connection destination management table 440, the name of the base where the display device 2 as the host device that can participate is installed and the IP address of the display device 2 as the host device are managed in association with each other. The connection destination management table 440 may not be provided.

However, in this case, the user of the participating device needs to input the IP address of the host device by an input device such as a touch panel in order to start remote request processing with the host device. Therefore, the user of the participating device obtains the IP address of the host device from the user of the host device by telephone, e-mail, or the like.

[Functional configuration of server section]
The server unit 90 is provided in each display device 2, and any display device 2 can play a role as a server unit. Therefore, the server unit 90 has a communication control unit 70 and a data management unit 80.

The communication control unit 70 controls communication with the communication control unit 70 in the client unit 20 in the same display device 2 and the communication control unit 70 in the client unit 20 in another display device 2 via the communication network 9. The data management unit 80 manages operation data, image data, and the like.

<Display device function for determining the margin color according to the aspect ratio of the image to be read and the display>
Subsequently, with reference to FIG. 13, the function of the display device 2 regarding the determination of the color of the margin 13 will be described. FIG. 13 is an example of a functional block diagram showing the function of the display device 2 regarding the determination of the color of the margin 13 according to the aspect ratio of the image 11 read by the display device 2 and the display 3. The display device of FIG. 13 has a function partially extracted from the functions shown in FIG. 6, a aspect ratio comparison unit 51, a display direction detection unit 52, and a color determination unit 53. It also has an image storage unit 55 and an aspect ratio storage unit 54.

The image storage unit 55 stores the image data of the image 11 displayed by the display device 2. The image 11 is stored in a file format, and the display device 2 displays the image 11 included in the file page by page. The user can display any page. The image storage unit 55 includes not only the USB memory 5 but also a server on the cloud, NAS (Network Attached Storage), and the like. The RAM 203 or SSD 204 of the display device 2 may correspond to this. Further, the image data stored in the mobile terminal such as a smartphone may be received by the short-range communication circuit 219 (Bluetooth (registered trademark), Wi-Fi, NFC) or the like. When the display device 2 of another base stores image data, the image storage unit 55 also includes the display device 2 of the other base.

The display device 2 may display an image 11 saved by the same display device 2, or may save an image 11 saved by a different display device 2. Further, when displaying an image 11 saved by the same display device 2 or a different display device 2, the display direction may be different between the time of saving and the time of displaying.

Further, the image 11 stored in the image storage unit 55 is not limited to the image 11 captured by the display device 2, and includes any image 11. For example, it may be an arbitrary PDF file, an image captured by a camera, an image transmitted by e-mail, an image downloaded from a network, or the like.

The operation processing unit 26 accepts the operation of reading the image 11 (file) performed by the user on the UI image, and notifies the file processing unit 40 of the selected file. The file processing unit 40 reads a file selected by the user from the image storage unit 55, and notifies the aspect ratio comparison unit 51 of the aspect ratio of the document data included in the file. In many cases, a plurality of pages of document data in one file have the same aspect ratio, but if they are different, the aspect ratio comparison unit 51 determines the aspect ratio for each page.

The aspect ratio can be calculated from the vertical and horizontal resolutions or sizes described in the file metadata. For example, the PDF file stores the vertical and horizontal sizes (millimeters) as metadata. In addition, image data formats include JPEG, TIFF, BMP, PNG, GIF, etc., but JPEG and TIFF are accompanied by metadata of the standard called EXIF (Exchangeable image file format). Vertical and horizontal resolutions or sizes are stored inside. Images in other formats also have vertical and horizontal resolutions or sizes in their metadata.

Therefore, the file processing unit 40 can detect the aspect ratio without reading the main body of the file. The aspect ratio may be described as it is in the metadata of the file.

Further, the display direction detection unit 52 detects the current display direction. FIG. 14 is a diagram illustrating a display direction. FIG. 14A shows a horizontal display direction, and FIG. 14B shows a vertical display direction. As shown in FIG. 14, the display 3 can be rotated at least 90 degrees. The rotation may be performed by the user or by the motor.

The display direction detection unit 52 detects whether the display direction rotated by the user is horizontal or vertical based on the detection result of the direction detection sensor 225. The display direction detection unit 52 notifies the aspect ratio comparison unit 51 of the detected display direction. The display 3 of the display device 2 may have a fixed direction (there is no mechanism for the user to rotate the display 3). In this case, the display direction is fixed to the horizontal direction or the vertical direction.

It will be described back to FIG. The aspect ratio comparison unit 51 compares the aspect ratio notified by the file processing unit 40 with the aspect ratio stored in the aspect ratio storage unit 54. As shown in Table 1, the aspect ratio storage unit 54 stores the aspect ratio in association with the display direction.

表１は、アスペクト比記憶部５４に記憶されているアスペクト情報の一例である。アスペクト情報は、ディスプレイ方向に対応づけられたディスプレイ３のアスペクト比を有している。例えば、縦向きの場合のアスペクト比は９（Ｗ）：１６（Ｈ）であり、横向きの場合のアスペクト比は１６（Ｗ）：９（Ｈ）である。

Table 1 is an example of the aspect information stored in the aspect ratio storage unit 54. The aspect information has the aspect ratio of the display 3 associated with the display direction. For example, the aspect ratio in the vertical orientation is 9 (W): 16 (H), and the aspect ratio in the horizontal orientation is 16 (W): 9 (H).

Since the aspect ratio of the display 3 is determined by the physical shape (vertical and horizontal resolution) of the display 3, when the entire display 3 is the display area of the image, the aspect information includes two aspect ratios as shown in Table 1. It only needs to be registered. However, for example, when the display device 2 displays the image 11 only on a part (display area) of the display 3, and the aspect ratio of the display area is different from the aspect ratio of the display 3, the aspect information is displayed. The aspect ratio for the area is stored. In this case, two aspect ratios are registered for each display area in association with the case where the display is in landscape orientation and portrait orientation.

Further, when the user can specify an arbitrary display area in the display device 2 (when the user can create a display area like a window), the aspect ratio comparison unit 51 has an aspect ratio of the display area from the OS (Operating System). Can be obtained.

The aspect ratio comparison unit 51 reads the aspect ratio associated with the display direction from the aspect ratio storage unit 54 and compares it with the aspect ratio notified by the file processing unit 40. The aspect ratio comparison unit 51 notifies the image generation unit 34 of the comparison result (match, mismatch) and the display direction. If the aspect ratios do not match, the aspect ratio comparison unit 51 sends a color determination request to the color determination unit 53.

When the display direction of the display device 2 is fixed to be horizontal or vertical, only one aspect ratio is stored in the aspect ratio storage unit 54, so that the aspect ratio comparison unit 51 uses the aspect ratio of the aspect ratio storage unit 54. The ratio is compared with the aspect ratio of the image 11.

The color determination unit 53 acquires the image 11 (one page in the file) from the file processing unit, and determines the color not used in the image 11 as the color of the margin 13. The color of the margin 13 is notified to the image generation unit 34.

When the comparison results match, the image generation unit 34 generates a background image by scaling the image 11 to the same size as the display area. If the comparison results do not match, the image 11 is placed at a predetermined position (for example, in the vertical direction or in the center in the horizontal direction). In this case, the image generation unit 34 scales the image 11 so that the long side of the image 11 matches the length of the corresponding side of the display area. Further, the image generation unit 34 displays the color of the margin 13 as the color determined by the color determination unit 53, with the area in which the image 11 is not displayed as the margin 13.

<How to determine the color>
Subsequently, a method of determining a color by the color determining unit 53 will be described with reference to FIG. FIG. 15 is a diagram illustrating the image 11 and the colors included. For convenience of explanation, the edge of the image 11 is shown by a dotted line, but this dotted line does not actually exist. The color of each pixel of the image 11 is determined by RGB. For example, white is (R, G, B) = (255,255,255), black is (R, G, B) = (0,0,0), and yellow is (R, G, B) = (255,255). , 0) and so on.

The color determination unit 53 extracts a color for each pixel included in the image 11 and identifies a color not used in the image 11. Assuming that the gradation of R, G, and B is 8 bits per pixel, the types of colors are about 16.7 million colors, so it is easy to find unused colors. The color determination unit 53 may delete the color used in each pixel of the image 11 from the list of 16.7 million colors and select a color from the rest.

Since it takes time to determine each pixel, the color determination unit 53 may perform color reduction processing on the image 11. In the color reduction processing, when reducing 256 × 256 × 256 colors to, for example, 4 × 4 × 4, the section from 0 to 255 is changed from 0 to 63, 64 to 127, 128 to 191, 192 for each of R, G, and B. Divide into 4 of 255. When the color determination unit 53 converts the values 0 to 255 of each pixel to 0 to 3, the color can be reduced to 64 colors.

Some representative colors (hereinafter referred to as representative colors) may be prepared in advance. For example, white, black, gray, red, blue, green, yellow, orange, pink, purple, brown, gold, and silver are typical colors. FIG. 16 is an example of a diagram showing representative colors in a palette. For example, the color determination unit 53 deletes the color used in each pixel of the image 11 from the list of 12 colors, and selects a color from the rest.

In addition, there may be a case where the color used in the image 11 is not included in the representative color. Therefore, among the representative colors, the color whose distance from the color used in the image 11 is within the threshold value is deleted. The distance between colors uses the Euclidean distance in a predetermined color space such as RGB color space, Lab color space, or XYZ color space as an index.

When the number of colors used in the image 11 is large, for example, when the image 11 uses all the representative colors, the color determination unit 53 includes a predetermined range including the edge of the image 11 (hereinafter, referred to as an edge portion). It is advisable to determine the unused color only from). Since it is the edge of the image 11 that comes into contact with the margin 13, if the color of the margin 13 is different from the color of the edge of the image 11, the user can distinguish the boundary between the image 11 and the margin 13.

FIG. 17 is a diagram illustrating colors not used at the edges of image 11. In FIG. 17A, since the image 11 is in the vertical direction and the display direction is in the horizontal direction, it is the vertical edge portion 14a of the image 11 that comes into contact with the margin 13. The color determining unit 53 determines an unused color only from the vertical edge portion 14a of the image 11.

In FIG. 17B, since the image 11 is horizontally oriented and the display direction is vertically oriented, it is the horizontal edge portion 14b of the image 11 that comes into contact with the margin 13. The color determining unit 53 determines an unused color only from the horizontal edge portion 14b of the image 11.

By doing so, even if the number of colors used in the image 11 is large, it becomes easy to find the unused colors from the representative colors. The method of determining the unused color from only the edges 14a and 14b may be adopted before determining whether or not the entire image 11 uses all the representative colors.

The edge portions 14a and 14b may be one pixel at the outermost edge of the image 11, but in order to clarify the boundary between the image 11 and the margin 13, the edge portion 14a has a width of about several tens to several hundred pixels from the outermost edge. , 14b may be used.

If it is not possible to determine an unused color from only the edges 14a and 14b, it is advisable to widen the range of representative colors. For example
(i) Determine the unused color from the complementary colors of the representative colors.
(ii) Determine the unused color from the mixture of the two representative colors.
And so on.

<Scale of image>
FIG. 18 is a diagram illustrating the scaling of the image 11. In FIGS. 18A and 18B, a case where the display 3 is oriented sideways will be described. That is, the image 11 is in the vertical orientation and the display 3 is in the horizontal orientation. The vertical size of the image 11 shown in FIG. 18A is H1, and the horizontal size is W1. Further, the vertical size of the display 3 shown in FIG. 18B is H2, and the horizontal size is W2. In order to fit the image 11 in the display 3, it is necessary to fit the longer side (H1 in the figure) of the image 11 in the display 3. Therefore, the image generation unit 34 scales the image 11 with H2 / H1 as the ratio.
Vertical size of image 11 after scaling H2 = H1 × H2 / H1
Horizontal size of image 11 after scaling Wa = W1 × H2 / H1
In FIGS. 18C and 18D, a case where the display 3 is oriented vertically will be described. That is, the image 11 is in landscape orientation and the display is in portrait orientation. The vertical size of the image 11 shown in FIG. 18C is H3, and the horizontal size is W3. Further, the vertical size of the display 3 shown in FIG. 18D is H4, and the horizontal size is W4. In order to fit the image 11 in the display 3, it is necessary to fit the longer side (W3 in the figure) of the image 11 in the display 3. Therefore, the image generation unit 34 scales the image 11 with W4 / W3 as the ratio.
Vertical size of image 11 after scaling Ha = H3 × W4 / W3
Horizontal size of image 11 after scaling W4 = W3 x W4 / W3
FIG. 19 is a diagram illustrating a method of determining the position of the margin 13. FIG. 19 describes a case where the image 11 is arranged in the center of the display 3. Therefore, the margins 13 are generated on the left, right, top and bottom of the image 11.

FIG. 19A shows a case where the display 3 is oriented sideways. Using the Wa calculated in FIG. 18, the positions of the margins 13 are the area from the left edge of the display to (W2-Wa) / 2 and the area from the right edge of the display to (W2-Wa) / 2.

FIG. 19B shows a case where the display is oriented vertically. Using Ha calculated in FIG. 18, the position of the margin 13 is the area from the upper end of the display to (H4-Ha) / 2 and the area from the lower end to (H4-Ha) / 2.

The image generation unit 34 displays the margin 13 whose position is determined in this way in the determined color. Although the position of the image 11 is set to the center of the display 3 in FIG. 19, the position of the margin 13 may be the left end or the right end of the sideways display 3 as shown in FIG. 19A. In a portrait display as shown in FIG. 19B, the upper end or the lower end may be used.

Further, in any of the cases of FIGS. 19A and 19B, the position of the margin 13 may be specified by the user. For example, select from "top", "bottom", and "center" displayed on the setting screen. Alternatively, as shown in FIGS. 19A and 19B, it is preferable that the display device 2 displays the image 11 in the center, for example, and then the user can move the image 11 by hand.

The image generation unit 34 does not always have to display the margin 13 in the determined color. For example, it may be displayed in a color determined by the color determining unit 53 on a regular basis. Further, the image generation unit 34 may display the margin 13 with a color determined by the color determination unit 53 according to a user operation such as double tapping the margin 13. Double tap again to restore the color.

Further, in the present embodiment, it has been described that the margin 13 is uniformly displayed in the color determined by the color determining unit 53, but only the outer periphery of the margin 13 may be displayed in the color determined by the color determining unit 53. Further, the color determined by the color determining unit 53 may be displayed linearly at the boundary between the margin 13 and the image 11. Further, the margin 13 may be shaded with a color determined by the color determining unit 53, or may be given a gradation.

<Operation procedure>
FIG. 20 is an example of a flowchart showing a procedure for determining the color of the margin 13 of the display area according to the aspect ratio of the image 11 read by the display device 2 and the display 3.

First, since the user performs the operation of reading the image 11, the operation processing unit 26 accepts the operation of reading the image 11 (file) (S1). The file processing unit 40 reads the designated image 11 (S2). As a result, the aspect ratio of the image 11 is detected.

On the other hand, the display direction detection unit 52 detects the current display direction (S3). The display orientation may be detected in real time, or the last detected display orientation may be retained. The display orientation set by the user may be used.

Next, the aspect ratio comparison unit 51 compares the aspect ratio of the image 11 with the aspect ratio read from the aspect ratio storage unit 54 according to the display direction (S4). For example, when the aspect ratio of the image 11 is W1: H1 and the aspect ratio read from the aspect ratio storage unit 54 is W2: H2,
W2 / H2-α <W1 / H1 <W2 / H2 + α
If it satisfies, the aspect ratio is the same, and if it does not satisfy, it is judged to be different.

α is a margin of error, and indicates a range in which the aspect ratios can be considered to be the same even if the aspect ratios do not exactly match. By providing a margin of error, it is possible to increase the chances of full-screen display. Note that α may be a few percent of H2 / W2, but may be arbitrarily set by the user, for example. For example, if W1: H1 is 9:16, W2: H2 is 4: 3, and α is 5%.
4 / 3-0.067 <16/9 holds, but 16/9> 4/3 + 0.067, so it is judged that the aspect ratios are different.

When the aspect ratios are different (No in S5), the color determination unit 53 determines a color not used in the image 11 (S6). For example, a color that is not used in the entire image 11 is searched from the representative color, if it is not found, a color that is not used in the edges 14a and 14b is searched from the representative color, and if it is not found, the range of the representative color is expanded. I will go.

Next, the image generation unit 34 arranges the image 11 read by the file processing unit 40 in the center of the display 3, for example (S7). The image 11 is scaled (enlarged or reduced) so that the longer side of the image 11 matches the length of the corresponding side on the display 3 side.

Further, the image generation unit 34 changes the color of the margin 13 to the color determined in step S6 (S8).

On the other hand, when the aspect ratios are the same (Yes in S5), the image generation unit 34 arranges the image 11 read by the file processing unit 40 on the entire surface of the display (S9). In this case as well, the image 11 is often scaled.

The image generation unit 34 superimposes the background image on the stroke image or the like and displays it on the display 3 (S10).

The process of FIG. 20 describes the process of the display device 2 for reading a file. However, when a plurality of display devices 2 perform communication for a remote conference as shown in FIG. 4, the image 11 is another communication terminal. Shared with. Therefore, the display device 2 that has read the file transmits the representative color determined in the process of FIG. 20 to the other display device 2 during the conference. As a result, the other display device 2 can clarify the boundary between the image 11 and the display area (margin 13) without determining the representative color.

<Main effect>
As described above, in the display device 2 of the present embodiment, the color of the margin 13 is determined according to the color used in the image 11, so that the user can use the boundary between the image 11 and the display area (margin 13). Can be easily identified.

One file may contain a plurality of pages, but when there are a plurality of pages having different aspect ratios of the image 11 and the display, the color of the margin 13 may change for each page in the method of the first embodiment (the color of the margin 13 may change for each page (). Inconsistent). As a user, the color of the area changes each time the page is changed, which may cause a sense of discomfort.

Therefore, in this embodiment, the display device 2 examines the colors used in the image 11 of each page before displaying the image 11 of each page included in the file, and sets the color not used in any page as the margin 13. Decide on the color of. When the display device 2 displays the image 11 of each page, the margin 13 is displayed in a color common to all pages.

In this embodiment, the hardware configuration diagram of FIG. 5 and the functional block diagram shown in FIG. 6 or FIG. 13 described in the above embodiment will be described as being applicable.

<Determining colors that are not used on multiple pages>
FIG. 21 is an example of a diagram illustrating a method of determining a color that is not commonly used on each page among the representative colors. In FIG. 21, a check mark is set for the representative color used in association with the page. Therefore, since the color with the check mark is used in the image 11, the color determination unit 53 may search for a color that is not used in all the pages. Since yellow and gold are not used in FIG. 21, the color determining unit 53 determines yellow or gold as the color of the margin 13 common to all pages.

<Operation procedure>
FIG. 22 is an example of a flowchart showing a procedure for determining the color of the margin 13 of the display area according to the aspect ratio of the image 11 read by the display device 2 and the display 3. In the description of FIG. 22, the difference from FIG. 20 will be described.

In FIG. 22, when the file processing unit 40 reads the file in step S2, the aspect ratio comparison unit 51 determines whether or not there are a plurality of images having different aspect ratios from the display in the file (S101).

When there are a plurality of images having different aspect ratios from the display 3 in the file, the color of the margin 13 may change for each page. Therefore, as shown in FIG. 21, the color determination unit 53 is not used on each page. The color is determined (S102). The page for determining the unused color may be only the page including the image 11 having an aspect ratio different from that of the display 3.

When there are not a plurality of images having an aspect ratio different from that of the display 3 in the file (when there is no page or only one page), the process of FIG. 20 of the first embodiment is executed.

The following is executed every time the page is switched (S103). When the file is read, it is judged that the page is switched. The subsequent processing flow is the same as in FIG. 22, but in step S6-2, the color determination unit 53 adopts a color that is not commonly used and is determined by the color determination unit 53 in step S102.

By doing so, even if there are a plurality of pages having different aspect ratios of the image 11 and the display 3 in the file, it is possible to suppress the color of the margin 13 from changing for each page.

When a plurality of display devices 2 communicate for a remote conference as shown in FIG. 4, the display device 2 that has read the file uses the representative color determined in step S102 during the conference as in the case of the first embodiment. Is transmitted to the display device 2. The other display device 2 displays the margin 13 in a representative color when displaying the image 11 having an aspect ratio different from that of the display 3.

In this embodiment, when the user writes on the outside of the image 11 (that is, the margin 13), the display device 2 that reduces the margin 13 according to the written range will be described. When the display device 2 captures or prints only the range of the image 11, the handwritten data outside the range of the image 11 can be saved or the image 11 and the handwritten data can be printed.

In this embodiment, the hardware configuration diagram of FIG. 5 and the functional block diagram shown in FIG. 6 or FIG. 13 described in the above embodiment will be described as being applicable.

In this embodiment, the image generation unit 34 reduces the margin 13 according to the coordinates of the handwritten stroke. Further, the page processing unit 37 takes in the handwritten data and the image 11 except for the margin 13.

FIG. 23 is a diagram illustrating a margin 13 reduced by the image generation unit 34 according to the coordinates of the stroke. FIG. 23 (a) shows a case where the display 3 is oriented horizontally, and FIG. 23 (b) shows a case where the display 3 is oriented vertically. In FIGS. 23A and 23C, the user handwritten "ABC", but a part of the handwriting is superimposed on the margin 13.

The image generation unit 34 reduces the margin 13 so that the handwritten data does not overlap with the margin 13. As shown in FIGS. 23 (b) and 23 (d), the handwritten data does not overlap with the margin 13 because the margin 13 becomes small.

The image generation unit 34 takes X and Y coordinates as shown, and when the display 3 is in landscape orientation _{, the margin 13 is set from the left end of the display 3 to the leftmost stroke coordinate X left} , and the right end to the rightmost end of the display 3. The margin 13 is set to the _right of the coordinate X of the stroke of. The margin 13 does not become larger than the original size (the range of the image 11 becomes smaller than the original size).

Similarly, the display 3 is in vertical orientation, as the margin 13 until the most upper end _on the coordinate Y of the stroke from the top of the display 3, the up-coordinate Y of _a most bottom end of the stroke from the bottom of the display 3 and the margin 13. In this case as well, the margin 13 does not become larger than the original size (the range of the image 11 becomes smaller than the original size).

The page processing unit 37 captures the extended portion 15 generated by reducing the range of the image 11 and the margin 13 and saves it as a captured image.

FIG. 24 is an example of a flowchart showing a procedure for changing the size of the margin 13 and capturing an image. The process of FIG. 24 starts after the margin 13 is displayed.

The image generation unit 34 determines whether or not a stroke has been added to the display 3 (S201). The stroke processing unit 32 notifies the addition of the stroke.

When a stroke is added, the image generation unit 34 reduces the margin 13 to the left end and the right end (when the display is oriented horizontally) or the upper end and the lower end (when the display is oriented vertically) of the coordinates of the stroke (S202).

The page processing unit 37 determines whether or not the capture operation of the image 11 has been input (S203). The import operation is notified from the operation processing unit 26. Not only the import operation but also the print operation may be detected.

When the capture operation is input, the page processing unit 37 captures the handwritten data and the image 11 with the range excluding the margin 13 as the capture range (S204). That is, a captured image in the captured range excluding the margin 13 is generated. In the case of a print operation, the display device 2 transmits the captured image to the printing device.

<Main effect>
According to this embodiment, the margin 13 is reduced according to the range written by the user, and the range excluding the margin 13 is captured. Therefore, the user can increase the range that can be written and the range that can be captured only by handwriting.

In this embodiment, a display device that displays by projecting an image will be described. In a device that projects an image such as a projector, it is determined whether or not the aspect ratio of the projected image or a part of the display area is the same as that of the image 11. The treatment of the color of the margin 13 may be the same as in Examples 1 to 3.

<< Another example of display device configuration 1 >>
Although the display device of the present embodiment is described as having a large touch panel, the display device is not limited to the one having a touch panel.

FIG. 25 is a diagram showing another configuration example of the display device. In FIG. 25, the projector 411 is installed on the upper side of the normal whiteboard 413. This projector 411 corresponds to a display device. The normal whiteboard 413 is not a flat panel display integrated with a touch panel, but a whiteboard that the user directly handwrites with a marker. The whiteboard may be a blackboard, and may be a flat surface large enough to project an image.

The projector 411 has an ultra-short focus optical system, and can project an image with little distortion from about 10 cm onto the whiteboard 413. This image may be transmitted from the PC 400-1 connected wirelessly or by wire, or may be stored in the projector 411.

The user writes on the whiteboard 413 using a dedicated electronic pen 2501. The electronic pen 2501 has, for example, a tip portion that emits light when the user presses it against the whiteboard 413 for handwriting and the switch is turned on. Since the wavelength of light is near infrared or infrared, it is invisible to the user's eyes. The projector 411 has a camera, captures a light emitting unit, analyzes the image, and identifies the direction of the electronic pen 2501. Further, the electronic pen 2501 emits sound waves together with light emission, and the projector 411 calculates the distance from the arrival time of the sound waves. The position of the electronic pen 2501 can be specified by the direction and the distance. A stroke is drawn (projected) at the position of the electronic pen 2501.

Since the projector 411 projects the menu 430, when the user presses a button with the electronic pen 2501, the projector 411 identifies the pressed button by the position of the electronic pen 2501 and the ON signal of the switch. For example, when the save button 431 is pressed, the stroke (set of coordinates) handwritten by the user is saved by the projector 411. The projector 411 stores handwritten information in a predetermined server 412, a USB memory 5, or the like. Handwritten information is stored page by page. Since the coordinates are saved instead of the image data, the user can re-edit them. However, in the present embodiment, since the operation command can be called by hand, the menu 430 does not have to be displayed.

<< Another example of display device configuration 2 >>
FIG. 26 is a diagram showing another configuration example of the display device. In the example of FIG. 26, the display device includes a terminal device 600, an image projection device 700A, and a pen motion detection device 810.

The terminal device 600 is wiredly connected to the image projection device 700A and the pen motion detection device 810. The image projection device 700A projects the image data input by the terminal device 600 onto the screen 800.

The pen motion detection device 810 communicates with the electronic pen 820 and detects the motion of the electronic pen 820 in the vicinity of the screen 800. Specifically, the electronic pen 820 detects coordinate information indicating a point indicated by the electronic pen 820 on the screen 800 and transmits the coordinate information to the terminal device 600.

The terminal device 600 generates image data of a stroke image input by the electronic pen 820 based on the coordinate information received from the pen motion detection device 810, and causes the image projection device 700A to draw the stroke image on the screen 800.

Further, the terminal device 600 generates superimposed image data showing a superimposed image in which the background image projected on the image projection device 700A and the stroke image input by the electronic pen 820 are combined.

<< Another example of display device configuration 3 >>
FIG. 27 is a diagram showing a configuration example of the display device. In the example of FIG. 27, the display device includes a terminal device 600, a display 800A, and a pen motion detection device 810.

The pen motion detection device 810 is arranged in the vicinity of the display 800A, detects coordinate information indicating a point indicated by the electronic pen 820A on the display 800A, and transmits the coordinate information to the terminal device 600. In the example of FIG. 27, the electronic pen 820A may be charged by the terminal device 600 via the USB connector.

The terminal device 600 generates image data of a stroke image input by the electronic pen 820A based on the coordinate information received from the pen motion detection device 810, and displays the image data on the display 800A.

<< Another example of display device configuration 4 >>
FIG. 28 is a diagram showing a configuration example of the display device. In the example of FIG. 28, the display device includes a terminal device 600 and an image projection device 700A.

The terminal device 600 performs wireless communication (Bluetooth (registered trademark), etc.) with the electronic pen 820B, and receives the coordinate information of the point indicated by the electronic pen 820B on the screen 800. Then, the terminal device 600 generates image data of the stroke image input by the electronic pen 820B based on the received coordinate information, and causes the image projection device 700A to project the stroke image.

Further, the terminal device 600 generates superimposed image data showing a superimposed image in which the background image projected on the image projection device 700A and the stroke image input by the electronic pen 820 are combined.

As described above, each of the above-described embodiments can be applied in various system configurations.

<Other application examples>
Although the best mode for carrying out the present invention has been described above with reference to examples, the present invention is not limited to these examples, and various modifications are made without departing from the gist of the present invention. And substitutions can be made.

For example, in the present embodiment, a display device that can be used as an electronic blackboard has been described, but the display device may be any display device as long as it can display an image, for example, digital signage.

Further, in the present embodiment, the electronic blackboard has been described as an example, but any information processing device having a touch panel can be suitably applied. Information processing devices equipped with a touch panel include, for example, PJs (Projectors), output devices such as digital signage, HUD (Head Up Display) devices, industrial machines, imaging devices, sound collectors, medical devices, network home appliances, etc. It may be a notebook PC (Personal Computer), a mobile phone, a smartphone, a tablet terminal, a game machine, a PDA (Personal Digital Assistant), a digital camera, a wearable PC, a desktop PC, or the like.

Further, in the present embodiment, the coordinates of the pen are detected by the method of detecting the coordinates of the pen tip with the touch panel, but the coordinates of the pen tip may be detected by ultrasonic waves. The pen emits ultrasonic waves together with light emission, and the display device 2 calculates the distance from the arrival time of the ultrasonic waves. The position of the pen can be specified by the direction and distance. The projector draws (projects) the trajectory of the pen as a stroke.

Further, the configuration examples shown in FIGS. 6 and 13 are divided according to the main functions in order to facilitate understanding of the processing by the display device 2. The present invention is not limited by the method of dividing the processing unit or the name. The processing of the display device 2 can be divided into more processing units according to the processing content. It is also possible to divide one processing unit so as to include more processing.

Further, each function of the embodiment described above can be realized by one or a plurality of processing circuits. Here, the "processing circuit" in the present specification is a processor programmed to execute each function by software such as a processor implemented by an electronic circuit, or a processor designed to execute each function described above. It shall include devices such as ASIC (Application Specific Integrated Circuit), DSP (digital signal processor), FPGA (field programmable gate array) and conventional circuit modules.

1 Image processing system 2 Display device 3 Display 11 Image

Japanese Unexamined Patent Publication No. 2018-061236

Claims (9)

A display device that displays the acquired image.
When the aspect ratio of the image and the aspect ratio of the display area in which the image is displayed are different, the color of the margin generated when the image is displayed in the display area is determined according to the color used in the image. Color determination part and
An image generation unit that displays the image in the display area and displays the margin in a color determined by the color determination unit.
A display device characterized by having. The display device according to claim 1, wherein the color determining unit determines a color that is not used in the image among predetermined representative colors as the margin color. The display device according to claim 1 or 2, wherein the color determining unit determines the color of the margin according to the color used at the edge of the image in which the image is in contact with the margin. .. A display orientation detection unit that detects whether the display on which the display area is displayed is landscape or portrait orientation,
An aspect ratio comparison unit that compares the aspect ratio between the display area and the image according to the orientation of the display detected by the display direction detection unit.
The display device according to any one of claims 1 to 3, wherein the display device has. When the display device acquires a file containing a plurality of pages,
If the file contains multiple images with different aspect ratios
The color determining unit determines, among the predetermined representative colors, at least a color not used in the plurality of images having different aspect ratios as the color of the margin common to the plurality of images having different aspect ratios. The display device according to any one of claims 1 to 4, wherein the display device is characterized by the above. It has a stroke processing unit that detects the position of the input means in contact with the display on which the display area is displayed and accepts the input of the stroke.
The display device according to any one of claims 1 to 5, wherein the image generation unit reduces the margin so that the stroke processing unit does not overlap with the coordinates of the stroke that has received the input. The display device according to claim 6, wherein the image and the stroke in a range excluding the margin are captured and the captured image is generated according to the operation of the user. It is a display method performed by a display device that displays the acquired image.
When the aspect ratio of the image and the aspect ratio of the display area in which the image is displayed are different, it occurs when the color determining unit displays the image in the display area according to the color used in the image. Steps to determine the color of the margins and
A step in which the image generation unit displays the image in the display area and displays the margin in the color determined by the color determination unit.
A display method characterized by having. A display device that displays the acquired image,
When the aspect ratio of the image and the aspect ratio of the display area in which the image is displayed are different, the color of the margin generated when the image is displayed in the display area is determined according to the color used in the image. Color determination part and
A program for displaying the image in the display area and functioning as an image generation unit for displaying the margin in a color determined by the color determining unit.

Images