JP2010276792A - Image display system, image display method, and portable terminal - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image display system that suppresses increase in overall cost of the system, allows an image to be pasted on a display with easy operation, and allows smooth processing even when a plurality of persons operate a portable terminal, and to provide an image display method and a portable terminal. <P>SOLUTION: The portable terminal 1 includes: a camera 4; an image specifying part 17 specifying an image to be pasted; a position calculation part 9 specifying the position to paste the image by using a position detecting image; a display parameter calculation part 10 specifying a tilt of the image on the basis of the tilt of the portable terminal 1, and specifying the size of the image by using the position detecting image; and an information transmitting part 8 transmitting data and information on the image to a server 2. The server 2 includes a screen data editing part 13 editing screen data on the basis of the data and information on the image, and a screen data transmitting part 14 transmitting the edited screen data to a display 3. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image display system, an image display method, and a mobile terminal that can display an image of a mobile terminal on a display device.

  In recent years, a display device such as a television has been increased in screen size, and it is possible to share the same information among a plurality of people on the spot. In addition, an image taken with a mobile terminal such as a mobile phone or a digital camera can be displayed on a personal computer, a TV, etc., and enjoyed by the family. Conventionally, in order to display an image stored in a portable terminal on a display device, the image is transferred via a temporary storage medium such as an email or an SD card, and an operation device such as a mouse or a remote control provided in the display device. Is used to select an image to be displayed and determine a position to display the image. In such a case, since it is necessary to use an operation device such as a mouse to place (paste) one or more images on the display device, a certain level of skill and operation that can handle the operation device. It was necessary to get used to. Furthermore, when an image is displayed with an inclination or when the display size is enlarged or reduced, a higher level of skill is required.

  Conventionally, as an image display system in response to such a request, a touch panel display device is brought into contact with a mobile terminal on which an image is displayed, and image data of the mobile terminal is transmitted to the display device when contact is detected. What is pasted is known (for example, refer to Patent Document 1). In this image display system, the display position and size of an image are controlled according to the contact state between the display device and the portable terminal.

JP 2007-316925 A

  However, in the image display system described above, the display device needs to be a touch panel type and needs to calculate a layout such as the display position and size of the image. It was necessary to add functions and parts that are not provided in general televisions and displays, such as a wireless communication unit and a display control unit for controlling the layout. Therefore, there is a problem that it is difficult to realize the system and the cost for the system increases. In addition, since the operator of the mobile terminal needs to approach the display device and bring the mobile terminal into contact with the display device, there is a problem that it takes time to paste the image. Furthermore, it is assumed that such a system for pasting an image of a portable terminal on a large-screen display device is used in situations where a plurality of people enjoy an image or a meeting is being held. However, in the image display system using the touch panel as described above, when a plurality of people paste images at the same time, it becomes necessary to perform contact detection at a plurality of locations, or the processing in the display device becomes complicated. . Therefore, there is a limit to the processing capacity of the control device that can be mounted on the display device, and the load becomes too large, so that when the image display system is used by a plurality of people, the processing cannot be performed smoothly. There was a problem.

  The present invention has been made to solve such a problem, and can reduce the cost of the entire system and can attach an image to a display device with a simple operation. An object of the present invention is to provide an image display system, an image display method, and a mobile terminal that enable smooth processing even when a terminal is operated.

  An image display system according to the present invention is an image display system including a mobile terminal, a server, and a display device. The mobile terminal is an imaging unit that images the display device, and a paste for pasting to a display screen of the display device. An image specifying means for specifying an image, a position specifying means for specifying the pasting position of the pasted image on the display screen based on the captured image taken by the imaging means, and a pasted image on the display screen based on the inclination of the mobile terminal The inclination specifying means for specifying the inclination of the image, the size specifying means for specifying the size of the pasted image on the display screen based on the captured image, the data on the pasted image, and the pasting position, tilt and size on the display screen of the pasted image Information transmitting means for transmitting information to the server, the server based on the data and information of the pasted image transmitted from the mobile terminal. Screen data editing means for editing the screen data of the pasted image displayed at a predetermined pasting position, inclination and size on the display screen, and screen data transmission for transmitting the screen data edited by the screen data editing means to the display device And means.

  According to the image display system of the present invention, the image specifying unit of the mobile terminal specifies the pasted image to be pasted on the display screen of the display device, while the imaging device captures the display device, thereby displaying the display screen. A captured image for specifying the pasting position and size of the pasted image inside is acquired. Then, the position specifying means specifies the paste position of the pasted image on the display screen based on the captured image, and the tilt specifying means specifies the tilt of the pasted image on the display screen based on the tilt of the mobile terminal, and the size The specifying unit specifies the size of the pasted image on the display screen based on the captured image. Further, the information transmission means transmits information about the pasted image data and the pasted position, inclination, and size on the display screen of the pasted image to the server. The server that has received the pasted image data and various information from the mobile terminal edits the screen data of the pasted image on the display screen based on the data and information from the mobile terminal by the screen data editing means, and the screen data transmitting means. The edited screen data can be transmitted to the display device. Accordingly, the display device can display the pasted image on the display screen according to the pasting position, inclination, and size specified on the mobile terminal side. As a result, for example, while viewing the pasted image to be pasted on the mobile terminal, the pasting position, tilt, and size can be determined directly based on the positional relationship with the display device and the tilt of the terminal itself. The operation can be performed intuitively and easily with the feeling of pinning a medium photo. Furthermore, since the pasting position, inclination, and size of the pasted image can all be specified on the mobile terminal side, and all the screen data editing on the display device can be performed on the server side, special processing is performed in the display device. Since it is unnecessary, a general television or display can be used as long as it can be connected to the network. As a result, the system can be constructed without using a special display device dedicated to the system, and the system can be realized without increasing the cost. Furthermore, since the pasting position of the pasted image is specified based on the captured image of the imaging means of the portable terminal, the pasting operation can be performed at a position away from the display device. Therefore, it is possible to perform the pasting without taking time and effort as compared with the touch panel type image display system. In addition, when a plurality of people paste a pasted image on each mobile terminal for one display device, the pasting position, inclination, and size of the pasted image are all specified in each mobile terminal, All editing of screen data is processed by a server having a high processing capacity. Therefore, according to the image display system according to the present invention, it is possible to perform processing smoothly compared to the case where the pasting position is specified or the screen data is edited in the display device having a limited processing capability. It is possible to improve the usability in a situation where a plurality of people enjoy images or a meeting held by a plurality of people. As described above, the cost of the entire system can be reduced, and an image can be pasted on the display device with a simple operation, and even when a plurality of people operate the mobile terminal, the process can be smoothly performed. it can.

  In the image display system according to the present invention, the position specifying unit specifies the pasting position based on the display content of the display screen at a predetermined position of the captured image of the imaging unit or the position of the display device in the captured image. It is preferable. When the display content of the display screen is such that the coordinates on the display screen can be specified, the relative positional relationship between the mobile terminal and the display device by referring to the display content at a predetermined position of the captured image of the imaging means Can be grasped. Further, by referring to the position of the display device in the captured image, it is possible to grasp the relative positional relationship between the mobile terminal and the display device. Accordingly, the pasting position of the pasted image can be easily specified based on the captured image.

  In the image display system according to the present invention, the mobile terminal further includes an acceleration sensor for detecting the tilt of the mobile terminal, and the tilt specifying means includes the tilt of the mobile terminal detected by the acceleration sensor and the pasting on the display screen. It is preferable to match the inclination of the image. The inclination of the portable terminal can be accurately grasped by the acceleration sensor, and the inclination of the pasted image can be easily specified by matching the output value with the slope of the pasted image.

  In the image display system according to the present invention, the size specifying means determines the size of the pasted image on the display screen based on the size of the display device in the captured image of the imaging means or the size of the display content on the display screen. It is preferable to specify. The relative distance between the portable terminal and the display device can be grasped according to the size of the display device and the size of display contents in the captured image. Accordingly, the size of the pasted image can be easily specified based on the captured image.

  The image display system according to the present invention preferably includes a plurality of display devices, and the screen data transmission means of the server transmits the screen data to the plurality of display devices. By transmitting screen data to a plurality of display devices, it is possible to share the same content at each location in a teleconference at a remote location.

  In the image display system according to the present invention, the image specifying unit specifies a pasted image that has already been pasted on the display screen via the imaging unit, and the position specifying unit is an imaging that is newly captured by the imaging unit. Based on the image, the pasting position of the pasted image is identified again, the tilt identifying unit again identifies the tilt of the pasted image based on the new tilt of the mobile terminal, and the size identifying unit newly captures the image using the imaging unit. It is preferable to specify the size of the pasted image again based on the captured image. As a result, the pasted image once pasted on the display device can be selected again and re-edited.

  Further, in the image display system according to the present invention, the mobile terminal further includes a drawing unit, the drawing unit specifies a position where drawing is performed on the display screen based on a movement trajectory with respect to the display screen of the mobile terminal, and the server The screen data editing means preferably edits the screen data so that drawing is performed at the position specified by the drawing means. For example, by selecting a color or pattern from the color palette on the terminal display of the mobile terminal and moving the mobile terminal toward the display screen, it is possible to draw a line or pattern on the display screen according to the movement trajectory.

  In the image display system according to the present invention, the portable terminal further includes a recording unit that records a voice, and the information transmission unit transmits the data of the pasted image associated with the voice recorded by the recording unit. It is preferable. By pasting the pasted image in association with the voice recorded by the recording means, for example, when the pasted image on the display screen is selected, the voice can be reproduced.

  The portable terminal according to the present invention displays an image based on an image capturing unit that captures an image of the display device, an image specifying unit that identifies a pasted image to be pasted on the display screen of the display device, and a captured image captured by the image capturing unit. Position specifying means for specifying the paste position of the pasted image on the screen, inclination specifying means for specifying the tilt of the pasted image on the display screen based on the tilt of the own terminal, and paste image on the display screen based on the captured image Size specifying means for specifying the size, and information transmitting means for transmitting information on the pasted image data and the pasted position, inclination, and size on the display screen of the pasted image to the server.

  According to the portable terminal according to the present invention, the image specifying unit specifies a pasted image to be pasted on the display screen of the display device, while the image capturing unit captures the display device, whereby the pasting position on the display screen or A captured image for specifying the size is acquired. Then, the position specifying unit specifies a pasting position of the pasted image on the display screen based on the captured image captured by the imaging unit, and the tilt identifying unit is configured to determine the pasted image on the display screen based on the tilt of the mobile terminal. The inclination is specified, and the size specifying means specifies the size of the pasted image on the display screen based on the captured image. Further, the information transmission means transmits the data of the pasted image to the server, and transmits information on the pasted position, inclination and size on the display screen of the pasted image to the server. Thus, the server that has received the pasted image data and various information from the mobile terminal edits the screen data of the pasted image displayed on the display screen based on the data and information, and sends the screen data to the display device. Can be sent. Furthermore, the display device can display the pasted image on the display screen according to the pasting position, inclination, and size specified on the mobile terminal side. As a result, for example, while viewing the pasted image to be pasted on the mobile terminal, the pasting position, tilt, and size can be determined directly based on the positional relationship with the display device and the tilt of the terminal itself. The operation can be performed intuitively and easily with the feeling of pinning a medium photo. In addition, the pasting position, inclination, and size of the pasted image can all be specified on the mobile terminal side, and the screen data on the display device can all be edited on the server side. Since it is unnecessary, a general television or display can be used as long as it can be connected to the network. As a result, the system can be constructed without using a special display device dedicated to the system, and the system can be realized without increasing the cost. Furthermore, since the pasting position of the pasted image is specified based on the captured image of the imaging means of the portable terminal, the pasting operation can be performed at a position away from the display device. Therefore, it is possible to perform the pasting without taking time and effort as compared with the touch panel type image display system. In addition, when a plurality of people paste a pasted image on each mobile terminal for one display device, the pasting position, inclination, and size of the pasted image are all specified in each mobile terminal, All editing of screen data is processed by a server having a high processing capacity. Therefore, according to the portable terminal according to the present invention, the processing can be smoothly performed as compared with the case where the pasting position is specified and the screen data is edited in the display device with limited processing capability. It is possible to improve the usability in situations where people enjoy images or in a meeting held by multiple people. As described above, the cost of the entire system can be reduced, and an image can be pasted on the display device with a simple operation, and even when a plurality of people operate the mobile terminal, the process can be smoothly performed. it can.

  An image display method according to the present invention is an image display method using a mobile terminal, a server, and a display device, in which the mobile terminal picks up an image of the display device, and the mobile terminal pastes it on the display screen of the display device. An image specifying step for specifying a pasted image to be attached, a position specifying step for specifying a pasting position of the pasted image on the display screen based on the captured image captured by the mobile terminal in the imaging step, and a mobile terminal An inclination specifying step for specifying the inclination of the pasted image on the display screen based on the tilt of the terminal, a size specifying step for specifying the size of the pasted image on the display screen based on the captured image, and a portable terminal , Information on the pasted image and information on the pasted position, tilt and size on the pasted image display screen to the server. And a screen data editing step in which the server edits the screen data of the pasted image displayed at a predetermined pasting position, inclination, and size on the display screen based on the pasted image data and information transmitted from the mobile terminal; The server includes a screen data transmission step of transmitting the screen data edited in the screen data editing step to the display device.

  According to the image display method of the present invention, the portable terminal identifies the pasted image to be pasted on the display screen of the display device in the image identification step, while the portable terminal images the display device in the imaging step. Then, a captured image for specifying the pasting position and size on the display screen is acquired. Then, the mobile terminal specifies the pasting position of the pasted image on the display screen based on the captured image captured in the imaging step in the position specifying step, and the display screen based on the tilt of the mobile terminal in the tilt specifying step. In the size specifying step, the size of the pasted image on the display screen is specified based on the captured image captured in the imaging step. Further, in the information transmission step, the portable terminal transmits information on the pasted image data and information on the pasted position, inclination, and size on the pasted image display screen to the server. The server that has received the paste image data and various information from the mobile terminal edits the screen data of the paste image displayed on the display screen based on the data and information from the mobile terminal in the screen data editing step. In the transmission step, the edited screen data can be transmitted to the display device. Accordingly, the display device can display the pasted image on the display screen according to the pasting position, inclination, and size specified on the mobile terminal side. In this way, for example, by visually observing the pasted image to be pasted on the portable terminal, the pasting position, tilt, and size can be directly determined by the positional relationship with the display device and the tilt of the terminal itself, so that a photo of a paper medium on a cork board or the like It can be operated intuitively and easily with the feeling of pinning. Furthermore, since the pasting position, inclination, and size of the pasted image can all be specified on the mobile terminal side, and all the screen data editing on the display device can be performed on the server side, special processing is performed in the display device. Since it is unnecessary, a general television or display can be used as long as it can be connected to the network. As a result, the system can be constructed without using a special display device dedicated to the system, and the system can be realized without increasing the cost. Furthermore, since the pasting position of the pasted image is specified based on the captured image at the imaging step of the portable terminal, the pasting operation can be performed at a position away from the display device. Therefore, it is possible to perform the pasting without taking time and effort as compared with the touch panel type image display system. In addition, when a plurality of people paste a pasted image on each mobile terminal for one display device, the pasting position, inclination, and size of the pasted image are all specified in each mobile terminal, All editing of screen data is processed by a server having a high processing capacity. Therefore, according to the image display method according to the present invention, it is possible to perform processing smoothly as compared with the case where the pasting position is specified and the screen data is edited in the display device with limited processing capability. It is possible to improve the usability in a situation where a plurality of people enjoy images or a meeting held by a plurality of people. As described above, the cost of the entire system can be reduced, and an image can be pasted on the display device with a simple operation, and even when a plurality of people operate the mobile terminal, the process can be smoothly performed. it can.

  According to the present invention, the cost of the entire system can be reduced, and an image can be pasted on a display device with a simple operation, so that even when a plurality of people operate a mobile terminal, the processing can be smoothly performed. It can be carried out.

It is a figure which shows the structure of the image display system which concerns on embodiment of this invention. It is a sequence diagram for demonstrating the flow of the control processing of the image display system which concerns on embodiment of this invention. It is a figure showing the sticking method of the sticking image displayed on a display apparatus. It is a figure for demonstrating the method to determine the inclination in the display screen of a sticking image, and a size. It is a figure for demonstrating re-editing of the pasted image already pasted.

  Hereinafter, preferred embodiments of an image display system, an image display method, and a mobile terminal according to the present invention will be described with reference to the drawings. In addition, in each figure, the same code | symbol is attached | subjected to the same element and the overlapping description is abbreviate | omitted.

  FIG. 1 is a diagram showing a configuration of an image display system 100 according to an embodiment of the present invention. As shown in FIG. 1, the image display system 100 includes a mobile terminal 1, a server 2, and a display device 3. The image display system 100 has a function of pasting an image in the portable terminal 1 on the screen of the display device 3 via the server 2. The mobile terminal 1, the server 2, and the display device 3 are connected to each other via a network NW.

  The mobile terminal 1 is composed of a camera-equipped mobile phone, and includes a camera (imaging unit) 4, a storage 5, an acceleration sensor 6, a terminal display 7, an information transmission unit (information transmission unit) 8, and a position calculation unit (position specification). Means) 9, display parameter calculation unit (tilt specifying unit, size specifying unit) 10, image processing unit 11, image specifying unit (image specifying unit) 17, drawing unit (drawing unit) 18, and recording unit (recording unit) 19. Have. The mobile terminal 1 is not limited to a camera-equipped mobile phone, and may be any portable terminal having a camera and a communication function, and may be a PHS or PDA with a camera, a digital camera with a communication function, or the like. Good.

  The camera 4 is an imaging device attached to the mobile terminal 1, and is provided on the back side of the terminal display 7, for example. The camera 4 normally captures an image while displaying an image on the terminal display 7. However, when the pasted image is pasted on the display device 3, the selected pasted image is displayed on the terminal display 7. Can continue to work in the ground. Thus, the camera 4 acquires a position detection image (captured image) for specifying the pasting position and size of the pasted image in a state where the operator can continue to confirm the pasted image on the terminal display 7. Can do. This position detection image is acquired at the timing when the operator presses the operation key with the camera 4 facing the display device 3. The camera 4 has a function of outputting the acquired position detection image to the position calculation unit 9 and the display parameter calculation unit 10. The camera 4 outputs the captured pasted image to the image specifying unit 17 when the pasted image selection mode for selecting the pasted image already pasted on the display screen 15 of the display device 3 is set. It has a function to do.

  The storage 5 is a storage medium built in the portable terminal 1 or a storage medium such as a memory card, and a plurality of images taken by the camera 4, images taken from an external device such as a PC or a digital camera, or a recording unit 19. It has a function to save the sound recorded in. Further, the storage 5 has a function of transmitting a stored image to the terminal display 7 or the image specifying unit 17 when the operator selects a pasted image. The acceleration sensor 6 is incorporated in the mobile terminal 1 and has a function of detecting a three-dimensional inclination of the mobile terminal 1. The acceleration sensor 6 has a function of outputting the detected detection value to the display parameter calculation unit 10. The terminal display 7 is a display provided in the mobile terminal 1 and has a function of displaying various information to the operator. The terminal display 7 displays a plurality of images in the storage 5 at the time of selecting the pasted image, and displays the identified pasted image while the camera 4 captures the position detection image.

  The information transmission unit 8 has a function of performing communication with the server 2 via the network NW, transmits the pasted image data to the server 2, and pastes the pasted position, inclination, and the like on the pasted image display screen. It has a function of transmitting information related to the size and audio data associated with the pasted image to the server 2. The information transmission unit 8 has a function of transmitting the drawing information specified by the drawing unit 18 to the server 2.

  The position calculation unit 9 has a function of specifying the pasting position of the pasting image on the display screen 15 based on the positional relationship between the mobile terminal 1 and the display device 3 acquired from the position detection image of the camera 4. Yes. Further, the position calculation unit 9 has a function of specifying the pasting position based on the display content of the display screen 15 at the center position of the position detection image or the position of the display device 3 in the position detection image. . When the display content of the display screen 15 is such that the coordinates on the display screen 15 can be specified, such as an initial screen consisting of numerals and alphabets shown in FIG. 3A, for example, the center position of the position detection image The relative positional relationship between the mobile terminal 1 and the display device 3 can be grasped by referring to the display content at. Further, the relative positional relationship between the mobile terminal 1 and the display device 3 can be grasped by referring to the position of the display device 3 in the position detection image. Accordingly, the pasting position of the pasted image can be easily specified based on the position detection image. The position calculation unit 9 can also specify a position using a pasted image already displayed on the display device 3 when a new image is pasted. Further, the position calculation unit 9 is not limited to the display content of the display screen 15 at the center position of the position detection image, and any position may be used as a reference. The position calculation unit 9 has a function of outputting the calculation result to the image processing unit 11.

  The display parameter calculation unit 10 is based on the tilt of the mobile terminal 1 and specifies the tilt of the pasted image on the display screen 15 and the mobile terminal 1 and the display device 3 acquired by the position detection image of the camera 4. The size of the pasted image on the display screen 15 is specified based on the relative distance. The display parameter calculation unit 10 has a function of matching the inclination of the mobile terminal 1 detected by the acceleration sensor 6 with the inclination of the pasted image on the display screen 15, and the size of the display device 3 in the position detection image of the imaging unit. Or, it has a function of specifying the size of the pasted image on the display screen based on the size of the display content on the display screen. Note that the inclination of the pasted image does not have to be completely coincident with the inclination of the mobile terminal 1 and may have an error in accordance with the display state on the display screen 15. The display parameter calculation unit 10 has a function of outputting the calculation result to the image processing unit 11.

  The image processing unit 11 edits the pasting position, inclination, and size based on the calculation results from the position calculation unit 9 and the display parameter calculation unit 10, for example, attached information of the pasted image such as the Exif option area of the JPEG image ( It has a function to be added to the information on the pasting position, inclination and size. The image processing unit 11 has a function of outputting the image data of the pasted image to the information transmitting unit 8 together with the attached information.

  The image specifying unit 17 has a function of specifying a pasted image to be pasted on the display screen 15 of the display device 3. The image specifying unit 17 can specify an image already stored in the storage 5 and selected by a key operation by the operator as a pasted image, or specify an image just captured by the camera 4 as a pasted image as it is. Furthermore, the pasted image already pasted on the display device 3 can be identified as a pastable image that can be re-edited by capturing the pasted image again with the camera 4 and capturing it by key operation.

  The drawing unit 18 has a function of drawing lines and patterns on the display screen 15 on which the pasted image has already been pasted. Specifically, the mobile terminal 1 is set to the drawing mode. In this case, the movement trajectory with respect to the display screen 15 of the mobile terminal 1 is acquired from the captured image from the camera 4, and the position on the display screen 15 is specified based on the movement trajectory. Enables drawing with. In the drawing mode, a color palette for selecting a color, a pattern, or the like is displayed on the terminal display 7, and a color or a pattern to be used for drawing can be selected. The drawing unit 18 has a function of outputting the specified drawing information to the information transmission unit 8.

  The recording unit 19 is composed of a recorder or the like built in the mobile terminal 1 and has a function of recording sound. The recording unit 19 can associate the voice and the pasted image by recording the voice with the pasted image specified by the image specifying unit 17, for example.

  The server 2 is physically configured as a computer system including a CPU, a main storage device RAM and ROM, an auxiliary storage device such as a hard disk, and a communication module that is a data transmission / reception device such as a network card. The server 2 includes an information receiving unit 12, a screen data editing unit (screen data editing unit) 13, and a screen data transmission unit (screen data transmission unit) 14.

  The information receiving unit 12 has a function of receiving various information transmitted from the information transmitting unit 8 of the mobile terminal 1 and outputting the received information to the screen data editing unit 13. The screen data editing unit 13 has a function of editing screen data of display contents displayed on the display screen 15 based on pasted image data and additional information. The screen data editing unit 13 has a function of editing the screen data so that drawing is performed at the position specified by the drawing unit 18 of the mobile terminal 1. Further, the screen data editing unit 13 has a function of editing an initial screen for specifying the coordinates of the display screen 15. The screen data transmission unit 14 has a function of transmitting the screen data edited by the screen data editing unit 13 to the display device 3.

  The display device 3 has a function of displaying images and videos on the display screen 15 and is configured by a generally used large display such as a large TV or a personal computer display that can be connected to the network NW. The display device 3 includes a display screen 15 and a screen data acquisition unit 16. The screen data acquisition unit 16 has a function of acquiring screen data from the server 2 via the network NW and displaying display contents based on the screen data on the display screen 15.

  Next, an image display method by the image display system 100 according to the embodiment of the present invention will be described with reference to FIGS. FIG. 2 is a sequence diagram for explaining the flow of control processing of the image display system 100 according to the embodiment of the present invention. FIG. 3 is a diagram showing a method for pasting a pasted image displayed on the display device 3, and FIG. 4 is a diagram for explaining a method for determining the inclination and size of the pasted image on the display screen 15.

  Before the control process of FIG. 2 is performed, the server 2 associates the ID of the display device 3 with the ID of the mobile terminal 1. Thereby, it is associated with which display device 3 the information is reflected when the portable terminal 1 is operated. As shown in FIG. 2, the screen data transmission unit 14 of the server 2 transmits the screen data of the initial screen to the display device 3 (step S1). For example, as shown in FIG. 3A, the initial screen is a screen on which characters and numerical values for specifying coordinates in the X-axis direction and the Y-axis direction on the display screen 15 of the display device 3 are described. In the example of FIG. 3A, the position in the X-axis direction is specified by alphabets “A, B, C...”, And the position in the Y-axis direction is specified by numerical values “1, 2, 3,. . Note that the process of S1 may be started in response to a key operation on the mobile terminal 1, or may be started by a request from the display device 3. The screen data acquisition unit 16 of the display device 3 acquires the screen data of the initial screen transmitted in S1, and displays the initial screen on the display screen 15 (step S2).

  On the other hand, in the portable terminal 1, an image to be pasted on the display device 3 is selected from among the images held in the storage 5 by an operator's key operation, and the image specifying unit 17 thereby selects the selected image. Is specified as a pasted image (step S3: image specifying step). The image specified in S3 is displayed on the terminal display 7. Next, as shown in FIG. 3, the operator moves the mobile terminal 1 while pointing the camera 4 of the mobile terminal 1 toward the display device 3 and performs a key operation. 3 (a), which is an image of the region indicated by the dotted line E), an image pasting position and a display size on the display screen 15 of the display device 3 are determined, and the acceleration during key operation is determined. The inclination of the pasted image is determined by the output value of the sensor 6 (step S4: imaging step). As shown in FIG. 3 (a), the pasting position of the image is determined by the relative positions of the mobile terminal 1 in the X-axis direction and the Y-axis direction with respect to the display device 3, and as shown in FIG. The inclination is determined by the inclination of the mobile terminal 1 in the three-dimensional direction, and the size of the image is determined by the relative distance between the mobile terminal 1 and the display device 3 as shown in FIG. In S4, while the mobile terminal 1 is being moved, the specified pasted image is displayed on the terminal display 7 of the mobile terminal 1, while the camera 4 continues to capture the display device 3, and the acceleration sensor 6 continues to detect the tilt of the mobile terminal 1. Then, at the timing determined by the key operation, the camera 4 transmits the acquired position detection image to the position calculation unit 9 and the display parameter calculation unit 10, and the acceleration sensor 6 displays the acquired tilt information as the display parameter calculation unit. 10 to send.

  Next, the position calculation unit 9 of the mobile terminal 1 calculates the pasting position of the pasted image based on the position detection image transmitted from the camera 4 in S4 (step S5: position specifying step). Specifically, the position calculating unit 9 calculates the positions in the X-axis direction and the Y-axis direction by identifying the characters and alphabets described in the range closest to the center point of the position detection image (FIG. 3 ( In the example of a), it is “G3”). Further, the display parameter calculation unit 10 of the mobile terminal 1 calculates the inclination of the pasted image based on the inclination information transmitted from the acceleration sensor 6 in S4 (step S6: inclination specifying step). Specifically, the display parameter calculation unit 10 calculates the inclination of the pasted image as the same value as the inclination of the mobile terminal 1 in the three-dimensional direction. In addition, the display parameter calculation unit 10 of the mobile terminal 1 calculates the size of the pasted image based on the position detection image transmitted from the camera 4 in S4 (step S7: size specifying step). Specifically, the size of the pasted image is calculated from the size of characters and alphabets (“G3” in the example of FIG. 3A) in the position detection image.

  Next, the image processing unit 11 of the mobile terminal 1 edits the pasting position, inclination, and size calculated in S5 to S7 and appends them to the attached information of the pasted image (for example, the Exif option area of the JPEG image) (step S8). ). The information transmitting unit 8 transmits the pasted image data to the server 2 together with the attached information edited in S8 (step S9: information transmitting step).

  The information receiving unit 12 of the server 2 receives the data of the pasted image and the attached information transmitted in S9, and the screen data editing unit 13 displays the data on the display screen 15 of the display device 3 based on the data and the attached information. The screen data for editing is edited (step S10: screen data editing step). In the process of S10, the screen data editing unit 13 considers the scale of the pasted image and the display screen 15, and calculates the position, inclination, and size of the pasted image on the display screen 15 based on the attached information. As for the display size, the larger the value calculated in S7 (that is, the closer the mobile terminal 1 is to the display device 3), the smaller the display size, and the smaller the value calculated in S7 (that is, the mobile terminal 1 is). The farther away from the display device 3, the larger the display size. Next, the screen data transmission unit 14 transmits the screen data edited in S10 to the display device 3 (step S11: screen data transmission step). The screen data acquisition unit 16 of the display device 3 receives the screen data transmitted in S11, erases the displayed initial screen, and based on the acquired screen data as shown in FIG. 3B. A screen is displayed (step S13). When the process of S13 ends, the process shown in FIG. 2 ends. The transmission of the screen data in S11 may be performed when there is an image received from the mobile terminal 1 in response to a periodic screen acquisition request from the display device 3, or the data is transmitted from the mobile terminal 1. It may be performed immediately after receiving and editing new screen data.

  In addition, after adding the process of FIG. 2, when adding a paste image further, an initial screen may be displayed again on the display screen 15 of the display apparatus 3, or the display content currently displayed on the display screen 15 The position and size of the new pasted image may be specified based on the above. In other words, the display contents that can be used as the position detection image are those output from the server 2 and can correspond to the display contents at each position on the screen and the coordinates of the display screen 15. Any display content may be used. Moreover, even if it is not an image which cannot be directly visually recognized, it may be an image whose position can be specified by performing a predetermined process such as an electronic watermark.

  When the pasted image is pasted on the display device 3 and the pasted image is re-edited, the portable terminal 1 is set to the re-editing mode by the operator's key operation. As shown in FIG. 5A, in the re-editing mode, the camera 4 captures a pasted image on the display screen 15, and the image identifying unit 17 identifies the pasted image captured as a re-editing pasted image. Next, as shown in FIG. 5B, when the operator moves or tilts the portable terminal 1 and performs a key operation at the determined position, the position calculating unit 9 attaches the re-edited paste image. The position is specified again, and the display parameter calculation unit 10 specifies the inclination of the reedited pasted image again and identifies the size of the reedited pasted image again. The image processing unit 11 processes the specified pasting position, inclination, and size again as additional information, and the information transmission unit 8 transmits the information to the server 2. The screen data editing unit 13 of the server 2 re-edits the screen data based on the reset additional information, and the screen data transmission unit 14 transmits the re-edited screen data to the display device 3. As a result, the re-edited pasted image is displayed on the display screen of the display device 3 as shown in FIG.

  Further, when drawing on the display screen 15 after the pasted image is pasted on the display device 3, the portable terminal 1 is set to the drawing mode by the operator's key operation. In the drawing mode, a color palette prepared in the application in the mobile terminal 1 is displayed on the terminal display 7, and the operator selects a color or pattern from the color palette. Thereafter, the operator points the camera 4 toward the display screen 15 of the display device 3 and moves the portable terminal 1 along a locus such that a desired line is drawn on the display screen 15. At this time, the camera 4 continues to acquire the captured image and continues to output the captured image to the drawing unit 18. The drawing unit 18 continues to specify the center position of the captured image as a part on the display screen 15 where a line is to be drawn. Thereby, the drawing unit 18 can specify the position where drawing is performed on the display screen based on the movement locus of the mobile terminal 1. Next, the information transmission unit 8 transmits the drawing information specified by the drawing unit 18 to the server 2. The screen data editing unit 13 of the server 2 edits the screen data so that a line is drawn with the selected color or pattern at the position specified by the drawing unit 18 of the mobile terminal 1, and the screen data transmission unit 14 The screen data on which the line is drawn is transmitted to the display device 3. Thereby, a line can be drawn on the display screen 15 in accordance with the movement locus of the mobile terminal 1. The drawing unit 18 may use not only the center position as a reference position but also other positions as reference positions when specifying.

  Further, when the pasted image is pasted on the display device 3, the voice recorded by the recording unit 19 can be associated with the pasted image. Specifically, after the pasted image is specified in S3, voice is recorded by the recording unit 19, and pasted on the display device 3 in a state where the identified pasted image and the voice are associated with each other. After pasting, the operator can play back the voice associated with the selected pasted image by selecting the pasted image on the display screen 15 with the camera 4.

  As described above, according to the image display system 100 and the mobile terminal 1 according to the present embodiment, the image specifying unit 17 of the mobile terminal 1 specifies the pasted image to be pasted on the display screen 15 of the display device 3. By imaging the display device 3 with the camera 4, a position detection image for specifying the relative position relationship and the relative distance between the mobile terminal 1 and the display device 3 and the pasting position and size is specified. get. And the position calculation part 9 pinpoints the sticking position of the sticking image in the display screen 15 based on the positional relationship of the portable terminal 1 acquired by the image for position detection of the camera 4, and the display apparatus 3, and calculates display parameter The unit 10 specifies the inclination of the pasted image on the display screen 15 based on the inclination of the mobile terminal 1, and sets the relative distance between the mobile terminal 1 and the display device 3 acquired by the position detection image of the camera 4. Based on this, the size of the pasted image on the display screen 15 is specified. Further, the information transmission unit 8 transmits the data of the pasted image to the server 2 and transmits information on the pasted position, inclination, and size on the pasted image display screen 15 to the server 2. Upon receiving the pasted image data and various information from the mobile terminal 1, the server 2 receives screen data of display contents displayed on the display screen 15 based on the data and information from the mobile terminal 1 in the screen data editing unit 13. The edited screen data can be transmitted to the display device 3 by the screen data transmission unit 14 after editing. Accordingly, the display device 3 can display the pasted image on the display screen 15 according to the pasting position, inclination, and size specified on the mobile terminal 1 side. In this way, it is possible to directly determine the pasting position, tilt, and size based on the positional relationship with the display device 3 and the tilt of the terminal itself while visually checking the pasted image to be pasted on the portable terminal 1, for example on a cork board. Operation can be performed intuitively and easily with the feeling of pinning a photo on a paper medium. Furthermore, since the pasting position, inclination, and size of the pasted image can all be specified on the mobile terminal 1 side, and all the screen data editing on the display device 3 can be performed on the server 2 side, Since special processing is unnecessary, a general television or display can be used as long as it can be connected to the network NW. As a result, the system can be constructed without using a special display device dedicated to the system, and the system can be realized without increasing the cost. Furthermore, since the pasting position of the pasted image is specified based on the position detection image of the camera 4 of the mobile terminal 1, the pasting operation can be performed at a position away from the display device 3. Therefore, it is possible to perform the pasting without taking time and effort as compared with the touch panel type image display system. In addition, when a plurality of people paste a pasted image on each mobile terminal 1 with respect to one display device 3, the pasting position, inclination, and size of the pasted image are all specified in each mobile terminal 1, All editing of screen data in the display device 3 is processed by the server 2 having high processing capability. Therefore, according to the image display system 100 and the portable terminal 1 according to the present invention, the processing is smoothly performed as compared with the case where the pasting position is specified and the screen data is edited in the display device 3 having a limited processing capability. Since it can be performed, it is possible to improve the usability in a situation where a plurality of people enjoy an image or a meeting held by a plurality of people. As described above, the cost of the entire system can be reduced, and an image can be pasted on the display device 3 with a simple operation, so that even when a plurality of people operate the mobile terminal 1, the process is smoothly performed. be able to.

  Further, the image display system 100 according to the present invention includes a plurality of display devices 3, and the screen data transmission unit 14 of the server 2 can transmit screen data to the plurality of display devices 3. By transmitting the screen data to a plurality of display devices 3, the same contents can be shared at each place in a video conference in a remote place.

  Further, according to the image display method according to the present embodiment, the portable terminal 1 identifies the pasted image to be pasted on the display screen 15 of the display device 3 in the image identifying step S3, while the portable terminal 1 captures the image. By capturing an image of the display device 3 in S4, a position detection image for specifying the pasting position and size based on the relative positional relationship and relative distance between the mobile terminal 1 and the display device 3 is acquired. Then, the portable terminal 1 identifies the pasting position of the pasted image on the display screen 15 based on the positional relationship between the portable terminal 1 and the display device 3 acquired from the position detection image of the camera 4 in the position identifying step S5. Then, in the inclination specifying step S6, the inclination of the pasted image on the display screen 15 is specified based on the inclination of the mobile terminal 1, and in the size specifying step S7, the mobile terminal 1 is acquired from the position detection image of the camera 4. The size of the pasted image on the display screen 15 is specified based on the relative distance between the display device 3 and the display device 3. Further, the portable terminal 1 transmits the data of the pasted image to the server 2 in the information transmission step S <b> 9 and transmits information about the pasting position, inclination and size on the display screen 15 of the pasted image to the server 2. The server 2 that has received the pasted image data and various information from the portable terminal 1 receives the screen data of the display content displayed on the display screen 15 based on the data and information from the portable terminal 1 in the screen data editing step S10. The edited screen data can be transmitted to the display device 3 in the screen data transmission step S11 after editing. Accordingly, the display device 3 can display the pasted image on the display screen according to the pasting position, tilt, and size specified on the mobile terminal 1 side. In this way, while visually checking the pasted image to be pasted on the portable terminal 1, the pasting position, tilt, and size can be determined directly by the positional relationship with the display device 3 and the tilt of the terminal itself, so It can be operated intuitively and easily as if you were pinning a photo. Furthermore, since the pasting position, inclination, and size of the pasted image can all be specified on the mobile terminal 1 side, and all the screen data editing on the display device 3 can be performed on the server 2 side, Since special processing is unnecessary, a general television or display can be used as long as it can be connected to the network NW. As a result, the system can be constructed without using a special display device dedicated to the system, and the system can be realized without increasing the cost. Further, since the pasting position of the pasted image is specified based on the position detection image in the imaging step S4 of the mobile terminal 1, the pasting operation can be performed at a position away from the display device 3. Therefore, it is possible to perform the pasting without taking time and effort as compared with the touch panel type image display system. In addition, when a plurality of people paste a pasted image on each mobile terminal 1 with respect to one display device 3, the pasting position, inclination, and size of the pasted image are all specified in each mobile terminal 1, All editing of screen data in the display device 3 is processed by the server 2 having high processing capability. Therefore, according to the image display method according to the present invention, it is possible to perform processing smoothly as compared with the case where the pasting position is specified and the screen data is edited in the display device with limited processing capability. It is possible to improve the usability in a situation where a plurality of people enjoy images or a meeting held by a plurality of people. As described above, the cost of the entire system can be reduced, and an image can be pasted on the display device with a simple operation, so that even when a plurality of people operate the mobile terminal 1, the processing can be performed smoothly. Can do.

  The present invention is not limited to the embodiment described above. For example, in the control process of FIG. 2 described above, screen data composed of alphabets and numbers as shown in FIG. 3A is transmitted as the initial screen transmitted in S1, but instead of this, a two-dimensional barcode is displayed. As described above, the screen data of the initial screen composed of identifiers randomly generated on the display screen 15 may be transmitted.

  In the control process of FIG. 2, the display content of the initial screen and the image with the pasted image already pasted are used for specifying the pasting position and size. Instead, the mobile terminal 1 and the display device are used instead. 3 may be used, the position and size of the display device 3 itself in the position detection image may be used.

  DESCRIPTION OF SYMBOLS 1 ... Portable terminal, 2 ... Server, 3 ... Display apparatus, 4 ... Camera (imaging means), 6 ... Acceleration sensor, 8 ... Information transmission part (information transmission means), 9 ... Position calculation part (position specification means), 10 ... display parameter calculation unit (tilt specifying means, size specifying means), 13 ... screen data editing part (screen data editing means), 14 ... screen data transmitting part (screen data transmitting means), 15 ... display screen, 17 ... image specifying Part (image specifying means), 18 ... drawing part (drawing means), 19 ... recording part (recording means), 100 ... image display system.

Claims (10)

An image display system including a mobile terminal, a server, and a display device,
The portable terminal is
Imaging means for imaging the display device;
Image specifying means for specifying a pasted image for pasting to the display screen of the display device;
Position specifying means for specifying the pasting position of the pasted image on the display screen based on the picked-up image picked up by the image pick-up means;
An inclination specifying means for specifying an inclination of the pasted image on the display screen based on an inclination of the mobile terminal;
Based on the captured image, size specifying means for specifying the size of the pasted image on the display screen;
Information sending means for sending information about the pasted image data and the pasted position, inclination and size of the pasted image on the display screen to the server,
The server
Screen data editing means for editing screen data of the pasted image displayed at a predetermined pasting position, tilt, and size on the display screen based on the data of the pasted image transmitted from the portable terminal and the information;
An image display system comprising: screen data transmission means for transmitting the screen data edited by the screen data editing means to the display device.   The position specifying means specifies the pasting position based on the display content of the display screen at a predetermined position of the captured image of the imaging means or the position of the display device in the captured image. The image display system according to claim 1. The mobile terminal further includes an acceleration sensor for detecting the tilt of the mobile terminal,
The image display system according to claim 1, wherein the inclination specifying unit matches the inclination of the portable terminal detected by the acceleration sensor with the inclination of the pasted image on the display screen.   The size specifying means specifies the size of the pasted image on the display screen based on the size of the display device in the captured image of the imaging means or the size of the display content of the display screen. The image display system according to any one of claims 1 to 3, wherein A plurality of display devices,
The image display system according to claim 1, wherein the screen data transmission unit of the server transmits the screen data to the plurality of display devices. The image specifying means specifies the pasted image that has already been pasted on the display screen via the imaging means,
The position specifying means specifies the pasting position of the pasted image again based on the captured image newly taken by the imaging means,
The inclination specifying means specifies again the inclination of the pasted image based on the new inclination of the mobile terminal,
The image display system according to claim 1, wherein the size specifying unit specifies the size of the pasted image again based on a captured image newly captured by the imaging unit. . The portable terminal further includes drawing means,
The drawing means specifies a position where drawing is performed on the display screen based on a movement trajectory of the mobile terminal with respect to the display screen,
The image display according to claim 1, wherein the screen data editing unit of the server edits the screen data so that drawing is performed at a position specified by the drawing unit. system. The portable terminal further includes recording means for recording voice,
The image display system according to claim 1, wherein the information transmission unit transmits the data of the pasted image associated with the voice recorded by the recording unit. Imaging means for imaging the display device;
Image specifying means for specifying a pasted image for pasting to the display screen of the display device;
Position specifying means for specifying the pasting position of the pasted image on the display screen based on the picked-up image picked up by the image pick-up means;
An inclination specifying means for specifying the inclination of the pasted image on the display screen based on the inclination of the terminal;
Based on the captured image, size specifying means for specifying the size of the pasted image on the display screen;
A portable terminal comprising: information transmission means for transmitting data on the pasted image data and information on the pasted position, inclination, and size of the pasted image on the display screen to the server. An image display method using a portable terminal, a server, and a display device,
An imaging step in which the mobile terminal images the display device;
An image specifying step for specifying a pasted image to be pasted on the display screen of the display device by the portable terminal;
A position specifying step for specifying a pasting position of the pasted image on the display screen based on the picked-up image picked up in the photographing step;
An inclination specifying step in which the mobile terminal specifies an inclination of the pasted image on the display screen based on an inclination of the mobile terminal;
A size specifying step in which the mobile terminal specifies the size of the pasted image on the display screen based on the captured image;
An information transmitting step in which the portable terminal transmits information on the pasted image data and information on the pasted position, inclination and size of the pasted image on the display screen;
Screen data for editing the screen data of the pasted image displayed at a predetermined pasting position, inclination, and size on the display screen based on the data of the pasted image transmitted from the portable terminal and the information. Editing steps,
An image display method comprising: a screen data transmission step in which the server transmits the screen data edited in the screen data editing step to the display device.

Images