JP2015192310A - Projection system, portable equipment, program, and control method of portable equipment - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a projection system capable of instructing a geometric correction to a projection image by operating portable equipment.SOLUTION: A projector includes: a display section for projecting a projection image on a projection object; a correction section for performing the geometric correction of the projected image in accordance with instruction information from a portable terminal 70; and a projection control section which makes the display section project a position identification image for identifying a position on the projection image and the projection image after the geometric correction. The portable terminal 70 includes: an imaging section 72 for capturing the projection image that is projected; a position identification section 802 for making the position on the projection image correspondent to a position on the captured image on the basis of the captured image obtained by capturing the position identification image; an operation input section 74 for detecting an operation onto the captured image; an instruction information generation section 804 for generating the instruction information represented by the position on the projection image in accordance with the correspondence by the position identification section 802; and a radio communication section 71 for transmitting the instruction information to the projector.

Description

  The present invention relates to a projection system, a portable device, a program, and a control method for the portable device.

A technique for performing geometric correction of a projected image in accordance with the shape of a projected object is known (see, for example, Patent Document 1).
Patent Document 1 discloses a video conversion device that geometrically corrects a video output from a video output device based on a test image projected on a screen taken by a camera-equipped mobile phone.

JP 2006-33357 A

However, in Patent Document 1, it is assumed that the entire projected image projected on the screen is captured and geometric correction is performed. Therefore, a part of the projected image is captured and the captured part is locally corrected. I can't. That is, even if a part of the projection image is photographed with a camera-equipped mobile phone and transmitted to the video conversion device, the video conversion device cannot recognize which part of the projection image may be corrected.
The present invention has been made in view of the above circumstances, and provides a projection system, a portable device, a program, and a method for controlling a portable device that can instruct geometric correction on a projected image by operating the portable device. With the goal.

In order to achieve the above object, a projection system of the present invention is a projection system including a projector and a portable device, and the projector includes a projection unit that projects a projection image on a projection object, and instruction information transmitted from the portable device. The correction unit that performs geometric correction of the projection image projected on the projection object, the position specifying image for specifying the position on the projection image, and the projection image after the geometric correction are displayed on the projection unit. The portable device is obtained by photographing the position specifying image and a photographing unit that photographs the position specifying image projected onto the projection object by the projection unit. A position specifying unit that associates a position on the projected image with a position on the captured image based on the captured image, and an operation on the captured image An instruction information generation unit that generates the instruction information represented by the position on the projection image from the operation content detected by the operation detection unit according to the association by the position specifying unit, the operation information to be detected, and the instruction information And a transmission device for transmitting to the projector.
According to the configuration of the present invention, it is possible to instruct geometric correction for a projected image by operating a portable device.

In the projection system according to the aspect of the invention, the imaging unit may capture a part of the position specifying image, and the position specifying unit may project the projection from a captured image obtained by capturing a part of the position specifying image. A position on the image is associated with a position on the captured image.
According to the configuration of the present invention, it is possible to capture a part of the projection image and instruct the projector where to perform geometric correction.

In the projection system according to the aspect of the invention, the projection control unit may alternately switch a correction target image to be the geometric correction target and the position specifying image as the projection image and project the projected image on the projection unit. It is characterized by making it.
According to the configuration of the present invention, the portable device can cause the portable device to capture the correction target image and the position specifying image, and can generate instruction information that represents the position on the projection image where the geometric correction is performed. it can.

In the projection system according to the aspect of the invention, the projector may notify the portable device of the shooting timing of the projection image based on the timing of switching between the correction target image and the position specifying image. The portable device includes a photographing control unit that causes the photographing unit to photograph the projection image in accordance with the photographing timing notified from the projector.
According to the configuration of the present invention, it is possible to notify the mobile device of the shooting timing of the correction target image and the position specifying image and cause the mobile device to take the correction target image and the position specifying image.

In the projection system according to the aspect of the invention, the projection control unit may project, as the projection image, a superimposed image obtained by superimposing the correction target image to be geometrically corrected and the position specifying image on the projection unit. The position specifying unit extracts the position specifying image from the captured image, and associates a position on the projection image with a position on the captured image.
According to the configuration of the present invention, it is possible to cause the portable device to shoot the correction target image and the position specifying image by one shooting, and it is possible to simplify the trouble of operation for performing geometric correction.
According to the configuration of the present invention, the position on the projection image can be accurately identified by the position specifying image.

In the projection system according to the aspect of the invention, the projection control unit may determine the position having a brightness higher than the second reference value in a region where the average brightness of the correction target image is lower than the first reference value. The superimposed image on which the image for use is superimposed is projected on the projection unit.
According to the configuration of the present invention, it is possible to improve the extraction accuracy of the position specifying image in the mobile terminal.

In the projection system according to the aspect of the invention, the projection control unit may specify a position having a luminance lower than the fourth reference value in a region where the average luminance of the correction target image is higher than the third reference value. The superimposed image on which the image for use is superimposed is projected on the projection unit.
According to the configuration of the present invention, it is possible to improve the extraction accuracy of the position specifying image in the mobile terminal.

In the projection system according to the aspect of the invention, the projector may change an interval between grid lines included in the position specifying image based on information received from the portable device.
According to this configuration, the operator of the mobile terminal does not need to pay attention to the shooting range.

In the projection system according to the aspect of the invention, the portable device may include a display unit that displays the captured image captured by the imaging unit.
According to this configuration, the photographed image photographed by the photographing unit can be displayed.

In the projection system according to the aspect of the invention, the display unit may capture the first captured image obtained by the photographing unit photographing the correction target image, and the photographing unit may photograph the position specifying image. Of the second captured images obtained in this manner, only the first captured image is displayed.
According to this configuration, the operator of the mobile terminal can specify the position on the projected image while viewing the display of the first captured image obtained by capturing the correction target image.

The portable device of the present invention includes a photographing control unit that causes a photographing unit to photograph a position specifying image for specifying a position on the projection image, which is projected by a projector that projects a projection image, and the position specifying image. Based on the captured image, a position specifying unit that associates a position on the projected image with a position on the captured image, an operation detection unit that detects an operation on the captured image, and an operation content detected by the operation detection unit. An instruction information conversion unit that converts the instruction information represented by the position on the projection image according to the association by the position specifying unit, and a transmission control unit that causes the projector to transmit the instruction information to the projector. Features.
According to the configuration of the present invention, it is possible to instruct geometric correction for a projected image by operating a portable device.

The program of the present invention is a program for causing a computer to shoot a position specifying image for specifying a position on the projection image, which is projected by a projector that projects a projection image, and the position specifying image. Based on the captured image, the association procedure for associating the position on the projected image with the position on the captured image, the detection procedure for detecting an operation on the captured image, and the operation content detected by the detection procedure, According to the association by the association procedure, a generation procedure for generating instruction information represented by a position on the projection image and a transmission procedure for causing the projector to transmit the instruction information to the projector are executed. .
According to the configuration of the present invention, it is possible to instruct geometric correction for a projected image by operating a portable device.

The method for controlling a portable device of the present invention includes a shooting step of shooting a position specifying image for specifying a position on the projection image, which is projected by a projector that projects a projection image, and the position specifying Based on the captured image of the image, an association step that associates a position on the projected image with a position on the captured image, a detection step that detects an operation on the captured image, and an operation content detected by the detection step A generation step of generating instruction information represented by a position on the projection image in accordance with the association in the association step; and a transmission step of causing the projector to transmit the instruction information to the projector. .
According to the configuration of the present invention, it is possible to instruct geometric correction for a projected image by operating a portable device.

  According to the present invention, it is possible to instruct geometric correction for a projected image by operating a portable device.

It is a functional block diagram of each part which comprises a projector. It is a figure which shows an example of the image for coordinate detection. It is a functional block diagram of each part which comprises a portable terminal. (A) shows a state where the first projection image is projected on the projection object, and (B) shows a state where the second projection image is projected on the projection object. It is a figure which shows an example of a correspondence table. (A) is a figure which shows a mode that drag-and-drop operation is performed, (B) is a figure which shows the image displayed after drag-and-drop operation. (A) is a figure which shows an example of a projection object, (B) is a figure which shows an example of the 1st projection image projected on a projection object. (A) is a figure which shows the state which projected the 1st projection image on the projection object, (B) is a figure which shows the imaging | photography range of a portable terminal. (A) is a figure which shows a mode that drag-and-drop operation is performed, (B) is a figure which shows the image displayed after drag-and-drop operation. (A) is a figure which shows the projection image and projection object before geometric correction, (B) is a figure which shows the projection image and projection object after geometric correction. It is a flowchart which shows the process sequence of a projector. It is a flowchart which shows the continuation of the process sequence of a projector. It is a flowchart which shows the process sequence of a portable terminal. It is a flowchart which shows the continuation of the process sequence of a portable terminal.

[First Embodiment]
Hereinafter, embodiments of a projection system will be described with reference to the accompanying drawings. First, the configuration of the projector 1 will be described with reference to FIG. FIG. 1 shows an example of the overall configuration of the projector 1. The projection system includes a projector 1 and a mobile terminal 70. The projector 1 is connected to an external image supply device 100 such as a personal computer or various video players, and projects an image based on input image data D (correction target image) input from the image supply device 100 onto a projection object. It is a device to display. Examples of the image supply device 100 include a video playback device, a DVD (Digital Versatile Disk) playback device, a TV tuner device, a CATV (Cable television) set-top box, a video output device such as a video game device, and a personal computer. The projected object may be an object that is not uniformly flat, such as a building or a model, or may have a flat projection surface such as a screen SC.
The mobile terminal 70 includes a smartphone, a mobile phone, a tablet PC, a PDA, and the like.

  The projector 1 includes an I / F (interface) unit 24 as an interface connected to the image supply apparatus 100. For the I / F unit 24, for example, a DVI interface, a USB interface, a LAN interface, or the like to which a digital video signal is input can be used. In addition, the I / F unit 24 includes, for example, an S video terminal to which a composite video signal such as NTSC, PAL, or SECAM is input, an RCA terminal to which a composite video signal is input, a D terminal to which a component video signal is input, or the like. Can be used. Furthermore, a general-purpose interface such as an HDMI connector conforming to the HDMI (registered trademark) standard can be used for the I / F unit 24. The I / F unit 24 may include an A / D conversion circuit that converts an analog video signal into digital image data, and may be connected to the image supply device 300 through an analog video terminal such as a VGA terminal. Note that the I / F unit 24 may perform transmission / reception of image signals by wired communication, or may perform transmission / reception of image signals by wireless communication.

  The projector 1 includes a display unit 10 that forms an optical image roughly and an image processing system that electrically processes an image displayed by the display unit 10. First, the display unit 10 will be described.

The display unit 10 includes a light source unit 11, a light modulation device 12, and a projection optical system 13.
The light source unit 11 includes a light source including a xenon lamp, an ultra high pressure mercury lamp, an LED (Light Emitting Diode), and the like. The light source unit 11 may include a reflector and an auxiliary reflector that guide light emitted from the light source to the light modulation device 12. The light source unit 11 is a lens group (not shown) for enhancing the optical characteristics of the projection light, a polarizing plate, a light control element that reduces the amount of light emitted from the light source on the path to the light modulation device 12, and the like. It may be provided.
The light modulation device 12 corresponds to a modulation unit that modulates light emitted from the light source unit 11 based on image data. The light modulation device 12 has a configuration using a liquid crystal light valve. The light modulation device 12 includes a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix, forms an image with the plurality of pixels, and modulates light emitted from a light source with the formed image. The light modulation device 12 is driven by the light modulation device driving unit 23, and forms an image by changing the light transmittance of each pixel arranged in a matrix.
The projection optical system 13 includes a zoom lens that performs enlargement / reduction of a projected image and a focus adjustment, a zoom adjustment motor that adjusts the degree of zoom, a focus adjustment motor that performs focus adjustment, and the like. The projection optical system 13 forms an image by projecting the light modulated by the light modulation device 12 onto a projection object using a zoom lens.

A projection optical system drive unit 21, a light source drive unit 22, and a light modulation device drive unit 23 are connected to the display unit 10.
The projection optical system drive unit 21 drives each motor included in the projection optical system 13 according to the control of the control unit 30. The light source driving unit 22 drives the light source included in the light source unit 11 according to the control of the control unit 30. The light modulation device driving unit 23 drives the light modulation device 12 according to the control of the control unit 30.

The projector 1 includes an image generation unit 56. The image generation unit 56 generates a coordinate detection image (position specifying image) according to an instruction from the control unit 30. The image generation unit 56 passes the generated coordinate detection image to the image processing unit 25. In the coordinate detection image, an image that can specify the position on the coordinate detection image is formed by performing image analysis on the coordinate detection image. FIG. 2 shows an example of a coordinate detection image. In the image for coordinate detection, vertical and horizontal lines (hereinafter referred to as grid lines) serving as a reference for specifying a position on the image are formed at predetermined intervals. In the following, the horizontal direction of the coordinate detection image is expressed as the X-axis direction, and the vertical direction is expressed as the Y-axis direction. In addition, the grid lines formed in the X-axis direction indicate the coordinate values of the grid lines in the Y-axis direction with numerals. Similarly, in the grid lines formed in the Y-axis direction, the coordinate values in the X-axis direction of the grid lines are indicated by numerals. For example, the position of the point P shown in FIG. 2 is (X, Y) = (1088, 544).
In FIG. 2, the coordinate values on the coordinate detection image are indicated by numerical values. However, if the coordinates can be specified, it is not necessary to indicate the numerical values. For example, the coordinate values may be represented by colors or graphic shapes.

The image processing system of the projector 1 is configured around a control unit 30 that controls the entire projector 1 in an integrated manner. The projector 1 includes a storage unit 54 that stores data processed by the control unit 30 and a control program executed by the control unit 30. In addition, the projector 1 includes a remote control light receiving unit 52 that detects an operation by the remote controller 5, and includes an input processing unit 53 that detects an operation via the operation panel 51 and the remote control light receiving unit 52.
The storage unit 54 is a non-volatile memory such as a flash memory or an EEPROM. The control unit 30 controls each unit of the projector 1 by reading and executing the control program stored in the storage unit 54. Further, the control unit 30 executes the functions of the projection control unit 31, the correction control unit 32, and the notification control unit 33 by executing the control program stored in the storage unit 54.

  The projection control unit 31 controls the projection optical system driving unit 21, the light source driving unit 22, the light modulation device driving unit 23, and the image processing unit 25 to project an image on a projection object. Further, the projection control unit 31 alternately projects an image based on the input image data D and an image based on the coordinate detection image generated by the image generation unit 56 onto the projection object every predetermined time. Hereinafter, an image projected on the projection object based on the input image data D is referred to as a first projection image. An image projected on the projection object based on the coordinate detection image is referred to as a second projection image. Further, the second projection image is projected onto the projection object with a size equal to or larger than that of the first projection image. That is, since the second projection image is an image for specifying the position of the first projection image, the second projection image is projected so as to coincide with the range on the projection object on which the first projection image is projected. It is preferable.

  For example, when receiving a start command for instructing the start of geometric correction from the portable terminal 70, the correction control unit 32 instructs the image generation unit 56 to generate a coordinate detection image. Further, the correction control unit 32 instructs the image processing unit 25 to perform geometric correction of the input image data D and the coordinate detection image based on the instruction information transmitted from the portable terminal 70.

  The notification control unit 33 generates an instruction command for instructing the mobile terminal 70 to perform shooting. The notification control unit 33 generates an instruction command based on the timing at which the image projected on the projection object is switched. The instruction command includes a first instruction command issued when the first projection image is projected on the projection object, and a second instruction command issued when the second projection image is projected on the projection object. Is included. The notification control unit 33 transmits the generated instruction command to the mobile terminal 70 via the wireless communication unit 55.

In the main body of the projector 1, an operation panel 51 including various switches and indicator lamps for operation by a user is disposed. The operation panel 51 is connected to the input processing unit 53. The input processing unit 53 appropriately lights or blinks the indicator lamp of the operation panel 51 according to the operation state and setting state of the projector 1 according to the control of the control unit 30. When the switch of the operation panel 51 is operated, an operation signal corresponding to the operated switch is output from the input processing unit 53 to the control unit 30.
The projector 1 also has a remote controller 5 that is used by the user. The remote controller 5 includes various buttons, and transmits an infrared signal corresponding to the operation of these buttons. The main body of the projector 1 is provided with a remote control light receiving unit 52 that receives an infrared signal emitted from the remote controller 5. The remote control light receiving unit 52 decodes the infrared signal received from the remote control 5, generates an operation signal indicating the operation content in the remote control 5, and outputs the operation signal to the control unit 30.

The image processing unit 25 acquires the input image data D in accordance with the control of the control unit 30, and the input image data D includes an image size, a resolution, a still image or a moving image, and a frame rate in the case of a moving image. Determine attributes and so on. The image processing unit 25 develops an image in the frame memory 26 for each frame. Further, the image processing unit 25 performs resolution conversion processing when the resolution of the acquired input image data D is different from the display resolution of the liquid crystal panel of the light modulation device 12. When zooming is instructed by operating the remote controller 5 or the operation panel 51, enlargement processing or reduction processing is performed, and the image after these processing is developed in the frame memory 26. Thereafter, the image processing unit 25 outputs the image for each frame developed in the frame memory 26 to the light modulation device driving unit 23 as a display signal.
Further, the image processing unit 25 includes a correction unit 251. The correction unit 251 performs geometric correction between the input image data D developed in the frame memory 26 and the coordinate detection image in accordance with an instruction from the correction control unit 32. Details of the geometric correction processing are disclosed in, for example, Japanese Patent Laid-Open No. 2013-78001. The image processing unit 25 passes the input image data D and the coordinate detection image after geometric correction to the light modulation device driving unit 23.

  In addition, the projector 1 includes a wireless communication unit 55. The wireless communication unit 55 includes an antenna (not shown), an RF (Radio Frequency) circuit, and the like, and communicates with the portable terminal 70 in accordance with a wireless communication standard under the control of the control unit 30. The projector 1 and the portable terminal 70 are connected so that various data can be transmitted and received by a wireless communication method. This wireless communication system includes, for example, a short-range wireless communication system such as a wireless local area network (LAN), Bluetooth (registered trademark), UWB (Ultra Wide Band), and infrared communication, and a wireless communication system using a mobile phone line. Can be adopted. The wireless communication unit 55 transmits the instruction command passed from the control unit 30 to the portable terminal 70 or receives the instruction information transmitted from the portable terminal 70 and passes it to the control unit 30.

Next, the configuration of the mobile terminal 70 will be described with reference to FIG.
The portable terminal 70 includes a wireless communication unit 71, an imaging unit 72, a display unit 73, an operation input unit 74, a storage unit 75, a control unit 80, and a memory 81, and these units are interconnected by a bus 90. In addition to the configuration shown in FIG. 3, the mobile terminal 70 includes a sound output speaker, a sound input microphone, a sound processing unit that performs sound processing on a sound signal input by the microphone and a sound signal output to the speaker, and the like. It may be.

  The wireless communication unit 71 includes an antenna (not shown), an RF (Radio Frequency) circuit, and the like, and communicates with the projector 1 in accordance with the wireless communication standard under the control of the control unit 80. The wireless communication unit 71 receives the instruction command transmitted from the projector 1 and passes it to the control unit 80, and transmits the instruction information passed from the control unit 80 to the projector 1.

  The imaging unit 72 performs imaging in accordance with the control of the control unit 80 and outputs the captured image data to the control unit 80.

  The display unit 73 includes a display panel 731 such as a liquid crystal display. The display unit 73 drives various drive circuits based on the control signal and the image signal input from the control unit 80 and displays an image on the display panel 731. The display panel is not limited to a liquid crystal display, and may be composed of other display devices such as an organic EL (electro-luminescence) display.

  The operation input unit 74 includes a touch sensor or the like that detects contact of an indicator such as a finger or a touch pen with the display unit 73. The touch sensor forms a touch panel by being integrally formed on the display panel 731 included in the display unit 73. The touch sensor may be various types of touch sensors such as a capacitance type, an ultrasonic type, a pressure sensitive type, a resistance film type, and a light detection type. The operation input unit 74 detects a position on the surface of the display panel 731 touched by the indicator as an input position, and outputs a position signal corresponding to the detected input position to the control unit 80.

  The storage unit 75 stores in a non-volatile manner control programs such as an operating system and application programs executed by the CPU of the control unit 81 and data processed by the control unit 81.

  The memory 81 is a storage device such as a ROM (Read Only Memory) or a RAM (Random Access Memory). The control unit 81 includes a CPU (Central Processing Unit: not shown), and controls the portable terminal 70 when the CPU executes an operating system stored in the storage unit 75. In addition, the control unit 81 executes various application programs stored in the storage unit 75 that is a nonvolatile storage device, so that the imaging control unit 801, the position specifying unit 802, the display control unit 803, and the instruction information generation unit. Each function of 804 is executed.

The imaging control unit 801 controls the imaging unit 72 according to the instruction command transmitted from the projector 1 and captures a projection image projected on the projection object.
As shown in FIG. 4A, when the first projection image is projected onto the projection object, the user operates the mobile terminal 70 so that a range in which the geometric correction of the first projection image is desired is captured. The imaging unit 72 is set to The imaging range of the imaging unit 72 may be a part or the whole of the first projection image, and may be a range that includes a range in which geometric correction is desired. Thereafter, the imaging control unit 801 controls the imaging unit 72 in accordance with the first instruction command transmitted from the projector 1 and images the set imaging range. Further, when a predetermined time has elapsed from the projection of the first projection image, the projection image projected onto the projection object is switched from the first projection image to the second projection image by the control of the projector 1 (see FIG. 4B). . If the user holds the portable terminal 70 so as not to move, the second projection image projected on the projection object is obtained by the imaging control unit 801 of the portable terminal 70 that has received the second instruction command transmitted from the projector 1. Shoot. The photographed image data obtained by photographing the second projection image is photographed within the same photographing range as the photographed image data obtained by photographing the first projection image.
When the first instruction command is input from the wireless communication unit 71, the imaging control unit 801 causes the imaging unit 72 to perform imaging, stores the captured image data in the memory 81, and the display control unit 803. To pass. The captured image data is displayed on the display panel 731 under the control of the display control unit 803. In the following, the image data captured when the first instruction command is received is referred to as first captured image data. The first captured image data is a first projection image, that is, an image obtained by projecting an image based on the input image data D onto a projection object.
In addition, when the second instruction command is input from the wireless communication unit 71, the shooting control unit 801 causes the imaging unit 72 to perform shooting and causes the memory 81 to store the captured image data. Hereinafter, the image data captured when the second instruction command is received is referred to as second captured image data. The second captured image data is an image obtained by projecting a second projection image, that is, an image based on the coordinate detection image onto the projection object.

The position specifying unit 802 performs image analysis on the second captured image data stored in the memory 81, and obtains the coordinate value of each lattice point on the second projection image.
On the coordinate detection image, as shown in FIG. 2, the coordinate values of each lattice point in the X-axis direction and Y-axis direction are indicated by numerals. The position specifying unit 802 performs coordinate analysis on the second photographed image data, analyzes the numbers recorded in the second photographed image data, and obtains the coordinate value of each lattice point on the second projection image. Hereinafter, the obtained coordinate values of the respective grid points are referred to as original image coordinate values. Also, the position specifying unit 802 performs processing for associating the original image coordinate values with the imaging coordinates. The imaging coordinates are a coordinate system of a solid-state imaging device included in the imaging unit 72. The solid-state imaging device has a structure in which a plurality of imaging devices (light receiving devices) are two-dimensionally arranged in the horizontal direction and the vertical direction. The imaging coordinates represent horizontal and vertical coordinates for identifying each light receiving element included in the solid-state imaging element. The position specifying unit 802 associates the original image coordinate value of the lattice point with a coordinate value (hereinafter referred to as an imaging coordinate value) at an imaging coordinate of an imaging element that has captured the lattice point. For example, the original image coordinate value of the point P shown in FIG. 4 is (X, Y) = (1088, 544), and the center of the point P is the coordinate (x, y) = (100, It is assumed that the image is captured by an image sensor located at 100). In this case, the imaging coordinates of the point P are (x, y) = (100, 100). The position specifying unit 802 creates a correspondence table by associating the original image coordinate values of the respective grid points with the coordinate values of the imaging coordinates. FIG. 5 shows an example of the correspondence table.
In the present embodiment, the original image coordinates are associated with the image capturing coordinates that specify the image sensor, but may be associated with the address of the memory 81 that stores the second captured image data.

  The display control unit 803 controls the display on the display panel 731 by outputting a control signal and an image signal to the display unit 73. Under the control of the display control unit 803, first captured image data obtained by capturing the first projection image is displayed on the display panel 731.

The instruction information generation unit 804 specifies the position on the second projection image, that is, the position indicated by the original image coordinate value, based on the position signal input from the operation input unit 74. This process will be specifically described.
The first photographed image data is displayed on the display panel 731 of the display unit 73 under the control of the display control unit 803. While viewing the first photographed image data, that is, the first projection image, the user designates a position where geometric correction is desired by a touch operation. The operation input unit 74 generates a position signal indicating the position on the display panel 731 touched by the user, and outputs the position signal to the control unit 80. For example, assume that the user performs a drag-and-drop operation on the display panel 731. In this case, the operation input unit 74 indicates a position signal indicating a position (hereinafter referred to as Q1) that the user first touched and a position indicating a position (hereinafter referred to as Q2) where the pointing device is released by a drop operation after dragging. Signal. Then, the operation input unit 74 outputs a signal indicating the drag and drop operation to the control unit 80 together with the generated position signals of Q1 and Q2. FIG. 6A shows a state in which the user performs a drag-and-drop operation on the first captured image data displayed on the display panel 731 of the mobile terminal 70. 6A, the state of the first captured image data displayed on the display panel 731 of the portable terminal 70 changes as shown in FIG. 6B.

  The instruction information generation unit 804 recognizes the touched position on the display panel 731 based on the input position signals Q1 and Q2. Next, the instruction information generation unit 804 specifies the shooting coordinate values of the pixels of the first captured image data displayed at each position on the display panel 731 indicated by the position signal, Q1 and Q2, respectively. By specifying the shooting coordinate value on the first shot image data, the shooting coordinate value on the corresponding second shot image data is specified. Since the first photographed image data and the second photographed image data are image data photographed without changing the setting of the imaging unit 72 such as the photographing range, the photographing coordinate value of the first photographed image data is specified. Thus, the captured image value of the corresponding second captured image data can be specified. Next, the instruction information generation unit 804 refers to the correspondence table shown in FIG. 5 and converts the identified shooting coordinate values into original image coordinate values. When the shooting coordinate value and the original image coordinate value corresponding to the position touched by the user are not registered in the correspondence table, the instruction information generation unit 804 stores the shooting coordinate value and the original image coordinate value recorded in the correspondence table. Are interpolated to obtain the corresponding original image coordinate values of Q1 and Q2. The instruction information generation unit 804 generates instruction information indicating the converted original image coordinate values of Q1 and Q2, and outputs the instruction information to the wireless communication unit 71. The instruction information is transmitted to the projector 1 by the wireless communication unit 71.

  A case where the first projection image shown in FIG. 7B is projected onto the projection object shown in FIG. 7A will be described. In addition, in order to distinguish a projection object and a 1st projection image, a projection object is shown as a continuous line and a 1st projection image is shown as a dotted line. FIG. 8A shows a state in which the first projection image is projected on the projection object. One of the vertices of the first projection image is projected with a deviation from the corresponding projection object. In order to correct the shift of the first projection image, the user adjusts the imaging unit 72 of the mobile terminal 70 so that the range in which the shift is to be corrected is included in the shooting range. FIG. 8B shows the shooting range of the imaging unit 72 with a solid line. The portable terminal 70 captures the first projection image and the second projection image projected on the projection object according to the first instruction command and the second instruction command transmitted from the projector 1. When the first projection image and the second projection image are captured, the mobile terminal 70 displays the first projection image on the display panel 731. In a state where the first photographed image data is displayed on the display panel 731, the user drags his / her finger to a position to be moved (Q1: movement source position) as shown in FIG. 9A and moves the dragged finger. Drop at the position to be moved (Q2: destination position). Thereby, on the display panel 731 of the portable terminal 70, as shown in FIG. 9B, the first projected image is corrected so that the image at the drag position moves to the drop position. In addition, the portable terminal 70 transmits the original image coordinate values of the drag position Q1 and the drop position Q2 to the projector 1. The projector 1 geometrically corrects the input image data D and the coordinate detection image based on the original image coordinate values received from the mobile terminal 70, and projects the geometrically corrected image onto the projection object. Thereby, as shown in FIG. 10 (A), from the state where one of the vertices of the first projection image is projected with a deviation from the vertex of the corresponding projection object, the first projection as shown in FIG. 10 (B). The vertex of the image is corrected to coincide with the vertex of the projection object.

Next, the processing procedure of the projector 1 will be described with reference to the flowcharts shown in FIGS.
First, the control unit 30 determines whether or not a start command for instructing the start of geometric correction is received from the portable terminal 70 (step S1). When the start command is received (step S1; Yes), the control unit 30 changes the operation mode to the geometric correction mode (step S2), and generates a coordinate detection image (step S3). Note that the process of step S3 is performed under the control of the image generation unit 56 of the control unit 30.

  Next, the control unit 30 projects the first projection image onto the projection object (step S4). When the first projection image is projected, the control unit 30 generates a first instruction command for instructing photographing of the first projection image, and transmits the first instruction command to the portable terminal 70 (step S5). Moreover, the control part 30 time-measures the elapsed time after projecting a 1st projection image on a projection object, and determines whether elapsed time passed predetermined time (step S6). If it determines with having passed predetermined time (step S6; Yes), it will replace with a 1st projection image, and the control part 30 will project a 2nd projection image on a projection object (step S7). When the second projection image is projected, the control unit 30 transmits a second instruction command for instructing photographing of the second projection image to the portable terminal 70 (step S8). When the second projected image is projected onto the projection object, the control unit 30 determines whether information transmitted from the mobile terminal 70 is received (step S9). When information is not received (step S9; No), the control part 30 determines whether the elapsed time after projecting a 2nd projection image passed predetermined time (step S10). When the predetermined time has not elapsed (step S10; No), the control unit 30 returns to step S9 and determines whether information transmitted from the portable terminal 70 has been received (step S9). When the predetermined time has elapsed (step S10; Yes), the control unit 30 returns to step S4 and again projects the first projected image onto the projection object (step S4). Note that the processes in steps S4 to S10 are performed by control by the projection control unit 31 of the control unit 30.

Next, a processing procedure when information is received from the mobile terminal 70 in step S9 will be described with reference to a flowchart shown in FIG.
When information is received from the portable terminal 70 (step S9; Yes), the control unit 30 determines whether the received information is instruction information (step S11). When the received information is the instruction information (step S11; Yes), the control unit 30 performs geometric correction of the input image data D and the coordinate detection image based on the original image coordinate values of Q1 and Q2 included in the instruction information. It performs (step S12). By performing geometric correction of both the input image data D and the coordinate detection image, the coordinate system of the input image data D and the coordinate detection image is prevented from differing. If the coordinate systems of the input image data D and the coordinate detection image are different, the correct coordinates of Q1 and Q2 cannot be obtained when geometric correction is performed next time. Note that the process of step S <b> 12 is performed under the control of the correction control unit 32 of the control unit 30. If the received information is not instruction information (step S11; No), the control unit 30 determines whether the received information is an end command for ending geometric correction (step S13). When the received information is an end command (step S13; Yes), the control unit 30 changes the operation mode from the geometric correction mode to the normal mode (step S14). In the normal mode, the first projection image and the second projection image are not alternately projected onto the projection object, but the first projection image based on the input image data D is projected onto the projection object. If the received information is not an end command (step S13; No), the control unit 30 performs a process according to the received information and proceeds to the process of step S4 shown in FIG.

Next, the processing procedure of the mobile terminal 70 will be described with reference to the flowcharts shown in FIGS.
First, the control unit 81 of the mobile terminal 70 determines whether or not an operation is input by the operation input unit 74 (step S21). When an operation is input (step S21; Yes), the control unit 81 determines whether or not the input operation is an operation for instructing the start of geometric correction (step S22). When the operation is a geometric correction start operation (step S22; Yes), the control unit 81 transmits a start command to the projector 1 (step S26). In addition, the control unit 81 causes the display unit 73 to display a message such as “Please set the shooting range of the camera so that the part for which geometric correction is to be performed is shot”. Thereafter, the control unit 81 determines whether or not an instruction command is received from the projector 1 (step S27). When the instruction command is received from the projector 1 (step S27; Yes), the control unit 81 determines whether or not the received command is the first instruction command (step S28). When the received command is the first instruction command (step S28; Yes), the control unit 81 captures the first projection image projected on the projection object under the control of the capturing control unit 801 (step S29). At this time, the first photographed image data photographed by the imaging unit 72 may be the entire first projection image projected on the projection object, or a range in the first projection image where geometric correction is desired. Some of them may be included. The control unit 81 stores the captured first captured image data in the memory 81 and causes the display panel 731 to display the first captured image data under the control of the display control unit 803.

  If it is determined in step S28 that the received command is not the first instruction command (step S28; No), the control unit 81 determines whether or not the received command is the second instruction command (step S30). ). When the control unit 81 determines that the received command is the second instruction command (step S30; Yes), the control unit 81 captures the second projection image projected on the projection object (step S31). The second photographed image data photographed at this time is preferably image data in the same range as the first photographed image. In other words, it is preferable that the camera of the portable terminal 70 is not moved as much as possible while the first captured image data and the second captured image data are captured. For this reason, a message such as “Please do not move the mobile terminal 70 as much as possible” may be displayed on the display unit 73 of the mobile terminal 70. Note that the processing in steps S26 to S31 is performed under the control of the imaging control unit 801 of the control unit 81.

  When the first captured image data and the second captured image data are captured (step S33), the control unit 81 calculates an original image coordinate value (step S34). Details of this processing will be described with reference to the flowchart shown in FIG. When the original image coordinate value is calculated, the control unit 81 generates instruction information including the calculated original image coordinate value. The control unit 81 transmits the generated instruction information to the projector 1 via the wireless communication unit 71 under the control of the instruction information generation unit 804 (step S35).

Next, a process for calculating the original image coordinate value will be described with reference to the flowchart shown in FIG.
First, the control unit 81 analyzes the second captured image data, and obtains the coordinate value of each lattice point on the captured second projection image (step S341). In the second projection image, as shown in FIG. 2, the coordinate values on the second projection image are recorded numerically. The position specifying unit 802 analyzes the numerical value captured in the second captured image data by image analysis, and obtains the coordinate value (original image coordinate value) of each lattice point on the second projection image. When the original image coordinate value of each grid point on the second projection image is obtained, the control unit 81 obtains the coordinate value (imaging coordinate value) of the image sensor that captured each identified grid point. The position specifying unit 802 associates the original image coordinate value of the specified lattice point with the imaging coordinate value and registers them in the correspondence table (step S342). Note that the processing of steps S341 and S342 is performed by control by the position specifying unit 802 of the control unit 81.

  Next, the control unit 81 displays the first photographed image data on the display panel 731 (step S343). This control is performed under the control of the display control unit 803.

  When the first photographed image data is displayed on the display panel 731, the control unit 81 monitors a signal output from the operation input unit 74 and determines whether or not a user operation input to the display panel 731 has been detected. (Step S344). The user designates a position where geometric correction is to be performed by a drag-and-drop operation while referring to the first captured image data displayed on the display panel 731.

  When the operation input is detected, the control unit 81 detects the position on the display panel 731 selected by the user. For example, when the user performs a drag-and-drop operation with a finger, the operation input unit 74 displays the position signal indicating the position Q1 touched first by the user and the position where the finger is released by the drop operation after dragging. A position signal indicating Q2 is output to the controller 81. The control unit 81 identifies the positions of Q1 and Q2 on the display panel 731 touched by the input position signal, identifies the pixels of the first photographed image data displayed at the identified positions Q1 and Q2, respectively, and performs photographing. A coordinate value is obtained (step S345). When the photographing coordinate value is obtained, the control unit 81 obtains the original image coordinate value on the second projection image with reference to the correspondence table shown in FIG. 5 (step S346). Since the first photographed image data and the second photographed image data are image data photographed without changing the setting of the imaging unit 72 such as a photographing range, the photographing coordinate value of the first photographed image data is set to the second photographing. It can be determined as a captured image value of the image data. The control unit 81 generates instruction information including the obtained original image coordinate value and outputs the instruction information to the wireless communication unit 71. Note that the processing in steps S344 to S346 is performed under the control of the instruction information generation unit 804. The instruction information is transmitted to the projector 1 by the wireless communication unit 71.

[Second Embodiment]
Next, a second embodiment will be described. In addition, since the structure of the projector 1 and the portable terminal 70 is a structure substantially the same as the structure of 1st Embodiment mentioned above, illustration and its description are abbreviate | omitted.
The projector 1 according to this embodiment does not alternately project the first projection image and the second projection image onto the projection object as in the first embodiment. In the present embodiment, the image generation unit 56 generates a coordinate detection image under the control of the correction control unit 32 and superimposes the generated coordinate detection image on the input image data D (hereinafter referred to as a superimposed image). Is generated. The image generation unit 56 passes the generated superimposed image to the image processing unit 25. The superimposed image is projected onto the projection object under the control of the projection control unit 31. In addition, since the first embodiment does not alternately project the first projection image and the second projection image, the projection control unit 31 instructs the portable terminal 70 to instruct the photographing timing and the second instruction command. Do not generate with command.

  The portable terminal 70 captures the whole or part of the projected image projected on the projection object at a predetermined rate by the imaging unit 72. The captured image data captured by the imaging unit 72 is displayed on the display panel 731 and stored in the memory 81. The position specifying unit 802 acquires captured image data of a superimposed image from the memory 81 and extracts a coordinate detection image by image analysis. Further, the position specifying unit 802 analyzes the extracted coordinate detection image to obtain the coordinates (original image coordinate value) of each lattice point of the coordinate detection image. In addition, the position specifying unit 802 specifies an imaging element that has captured each lattice point of the captured image data, and obtains an imaging coordinate value that is a coordinate value in the coordinate system of the imaging element. The position specifying unit 802 associates the original image coordinate value of the specified grid point with the imaging coordinate value and registers them in the correspondence table. Since the subsequent processing is the same as that of the first embodiment described above, description thereof is omitted.

(Modification)
The image generation unit 56 may divide the input image data D into predetermined areas and calculate the average luminance of each divided area. The image generation unit 56 compares the calculated average brightness of each region with the first reference value. The first reference value is a reference value used to determine whether the average luminance (gradation value) is low (that is, whether the area is dark). The first reference value is preferably a gradation value that is not more than half of the maximum gradation value, and can be, for example, a gradation value that is 25% of the maximum gradation value. The image generation unit 56 superimposes a coordinate detection image having a luminance higher than the second reference value on an area of the input image data D whose average luminance is lower than the first reference value. The second reference value is preferably sufficiently different from the first reference value so that the coordinate detection image can be easily recognized. The second reference value is preferably a gradation value that is half or more of the maximum gradation value. For example, the gradation value is 90% of the maximum gradation value, or the maximum gradation value (= all white). can do. That is, as the second reference value, a reference value including a range corresponding to high luminance including all white can be used.
In addition, the image generation unit 56 compares the calculated average brightness of each region with the third reference value. The image generation unit 56 superimposes a coordinate detection image having a luminance lower than the fourth reference value in a region where the average luminance of the input image data D is brighter than the third reference value. The third reference value is a reference value used to determine whether the average luminance (gradation value) is high (that is, whether the area is bright). The third reference value is preferably a gradation value that is half or more of the maximum gradation value, and can be, for example, a gradation value that is 75% of the maximum gradation value. The fourth reference value preferably has a sufficient difference from the third reference value so that the coordinate detection image can be easily recognized. The gradation value is preferably half or less of the maximum gradation value. For example, the gradation value may be 10% of the maximum gradation value or the minimum gradation value (= all black). That is, for the fourth reference value, for example, a reference value including a range corresponding to low luminance including all black can be used. When the projector 1 generates such a superimposed image and projects it on the screen SC, the mobile terminal 70 can easily extract the pattern of the coordinate detection image.

As described above, the system according to the first embodiment and the second embodiment of the present invention includes the projector 1 and the mobile terminal 70. The projector 1 includes a display unit 10, a correction unit 251, and a projection control unit 31 of the control unit 30. The display unit 10 projects a projection image on the projection object. The correction unit 251 performs geometric correction of the image projected on the projection object in accordance with the instruction information transmitted from the mobile terminal 70. The projection control unit 31 causes the display unit 10 to project a position specifying image for specifying a position on the projection image and a projected image after geometric correction.
The mobile terminal 70 includes an imaging unit 72, an operation input unit 74, and a control unit 80 that functions as a position specifying unit 802 and an instruction information generation unit 804. The imaging unit 72 captures the position specifying image projected on the projection object by the display unit 10. The position specifying unit 802 of the control unit 80 determines the position on the projection image and the position on the second projection image based on the shot image obtained by shooting the position specifying image among the shot images of the imaging unit 72. Associate. The operation input unit 74 detects an operation on the captured image. The instruction information generation unit 804 generates instruction information represented by the position on the projection image from the operation content detected by the operation input unit 74 in accordance with the association by the position specifying unit 802. The wireless communication unit 71 transmits instruction information to the projector 1. Accordingly, it is possible to instruct geometric correction on the projected image by operating the portable device 70.

  In addition, the imaging unit 72 captures a part of the second projection image, and the position specifying unit 802 includes a position on the projection image and a position on the captured image from the captured image obtained by capturing a part of the second projection image. Associate. Therefore, a part of the second projection image can be photographed, and a location for performing geometric correction can be instructed to the projector 1.

  In addition, the projection control unit 31 of the projector 1 alternately switches between the first projection image and the second projection image that are targets for geometric correction as the projection image, and causes the projection optical system 13 to project the projection image onto the projection object. Therefore, it is possible to cause the portable terminal 70 to capture the first projection image and the second projection image, and to generate instruction information that represents the position on the first projection image where the geometric correction is performed.

  In addition, the projector 1 includes a notification control unit 33 that notifies the portable terminal 70 of the shooting timing of the projection image based on the timing of switching between the first projection image and the second projection image. In addition, the mobile terminal 70 includes a shooting control unit 801 that causes the imaging unit 72 to capture a projection image in accordance with the shooting timing notified from the projector 1. Therefore, it is possible to cause the portable terminal 70 to capture the first projection image and the second projection image, and to generate the instruction information represented by the position on the first projection image that indicates the position where the geometric correction is performed.

  Further, the projection control unit 31 causes the projection optical system 13 to project an image obtained by superimposing the first projection image to be geometrically corrected and the second projection image as a projection image. The position specifying unit 802 extracts the second projection image from the captured image and associates the position on the projection image with the position on the captured image. Therefore, the first projection image and the second projection image can be photographed by the portable terminal 70 by one photographing, and the labor of the operation for performing geometric correction can be simplified.

  Further, the projector 1 changes the interval between the grid lines of the second projection image based on the information received from the mobile terminal 70. Therefore, the operator of the mobile terminal 70 does not need to pay attention to the shooting range.

In addition, the mobile terminal 70 includes a display unit 73 that displays a captured image captured by the imaging unit 72. Therefore, the captured image captured by the imaging unit 72 can be displayed on the display unit 73.
The display unit 73 includes a first captured image obtained by the imaging unit 72 capturing the first projection image, and a second captured image obtained by the imaging unit 72 capturing the second projection image. Of these, only the first photographed image is displayed. Therefore, the operator of the mobile terminal 70 can specify the position on the projected image while viewing the display of the first captured image obtained by capturing the first projected image.

In the above embodiment, the image display device (projector 1) includes a display unit (projection optical system 13) for displaying an image and a communication unit (wireless communication unit 55) capable of data communication with an external device (mobile terminal 70). ). Further, the image display device determines the timing at which the external device captures the display image based on the timing at which the display unit displays the image, and transmits data indicating the determined capturing timing to the external device through the communication unit. A control unit (notification control unit 33). The external device shoots the photographing unit (imaging unit 72), the communication unit (wireless communication unit 71) that receives data transmitted by the image display device, and the photographing unit at a timing specified by the data received by the communication unit. A shooting control unit (shooting control unit 801).
With this configuration, it is possible to cause an external device to perform shooting in accordance with the timing at which the image display device displays an image. For example, when the image display apparatus switches and displays the images, a specific image can be captured among the plurality of images displayed by switching.
Here, the data is processed based on the captured image captured by the imaging unit at the timing specified by the received data, and data for controlling the image display device is generated based on the result of the data processing. You may transmit to an image display apparatus.

The embodiment described above is a preferred embodiment of the present invention. However, the present invention is not limited to this, and various modifications can be made without departing from the scope of the present invention.
For example, when photographing the first and second photographed image data, the mobile terminal 70 may transmit information regarding the angle of view (zoom) set in the camera (imaging unit 72) to the projector 1. The projector 1 changes the grid line interval so that the number of grid lines in the captured image falls within a certain range. That is, if the number of grid lines in the captured image is larger than a predetermined number, an instruction to increase the grid line interval is issued, and if the number of grid lines is small, an instruction to decrease the grid line interval is issued. This eliminates the need for the user to be particularly concerned about the imaging range.

  The application program stored in the storage unit 75 of the mobile terminal 70 may be acquired by downloading from a server device on the network. Further, what is stored in an external storage device such as a USB (Universal Serial Bus) memory may be read into the portable terminal 70 and installed.

  DESCRIPTION OF SYMBOLS 1 ... Projector, 5 ... Remote control, 10 ... Display part (projection part), 13 ... Projection optical system, 25 ... Image processing part, 26 ... Frame memory, 30 ... Control part, 31 ... Projection control part, 32 ... Correction control part 33 ... Notification control unit, 54 ... Storage unit, 55 ... Wireless communication unit, 56 ... Image generation unit, 70 ... Mobile terminal, 71 ... Wireless communication unit, 72 ... Imaging unit (imaging unit), 73 ... Display unit, 74 ... operation input unit (operation detection unit), 75 ... storage unit, 80 ... control unit, 81 ... memory, 251 ... correction unit, 731 ... display panel, 801 ... photographing control unit, 802 ... position specifying unit, 803 ... display control Unit, 804 .. Instruction information generation unit.

Claims (13)

A projection system comprising a projector and a portable device,
The projector is
A projection unit that projects a projection image onto a projection object;
According to the instruction information transmitted from the portable device, a correction unit that performs geometric correction of the projected image projected on the projection object;
A position control image for specifying a position on the projection image, and a projection control unit that projects the geometrically corrected projection image on the projection unit,
The portable device is
An imaging unit that captures the position specifying image projected onto the projection object by the projection unit;
Based on a captured image obtained by capturing the position specifying image, a position specifying unit that associates a position on the projection image with a position on the captured image;
An operation detection unit for detecting an operation on the captured image;
An instruction information generating unit that generates the instruction information represented by the position on the projection image from the operation content detected by the operation detecting unit according to the association by the position specifying unit;
A projection system comprising: a transmission device that transmits the instruction information to the projector. The photographing unit photographs at least a part of the position specifying image,
The projection system according to claim 1, wherein the position specifying unit associates a position on the projection image with a position on the shot image from a shot image obtained by shooting at least a part of the position specifying image. .   3. The projection control unit according to claim 1, wherein the projection control unit alternately switches between a correction target image to be subjected to geometric correction and the position specifying image, and causes the projection unit to project the projection image. The projection system described. The projector includes a notification control unit that notifies the portable device of the shooting timing of the projection image based on the timing of switching between the correction target image and the position specifying image.
The projection system according to claim 3, wherein the portable device includes a photographing control unit that causes the photographing unit to photograph the projection image in accordance with the photographing timing notified from the projector. The projection control unit causes the projection unit to project a superimposed image in which a correction target image to be subjected to geometric correction and the position specifying image are superimposed as the projection image,
The projection system according to claim 1, wherein the position specifying unit extracts the position specifying image from the captured image and associates a position on the projection image with a position on the captured image.   The projection control unit projects the superimposed image obtained by superimposing the position specifying image having a luminance higher than a second reference value on a region where the average luminance of the correction target image is lower than a first reference value. The projection system according to claim 5, wherein the projection system is projected onto a projection unit.   The projection control unit projects the superimposed image obtained by superimposing the position specifying image having a luminance lower than a fourth reference value on a region where the average luminance of the correction target image is higher than a third reference value. The projection system according to claim 5, wherein the projection system is projected onto a projection unit.   The projection system according to claim 1, wherein the projector changes an interval between grid lines of the position specifying image based on information received from the portable device.   The projection system according to claim 1, wherein the portable device includes a display unit that displays the captured image captured by the imaging unit.   The display unit includes a first photographed image obtained by photographing the correction target image by the photographing unit and a second photographed image obtained by photographing the position specifying image by the photographing unit. 10. The projection system according to claim 9, wherein only the first photographed image is displayed. An imaging control unit that causes the imaging unit to image a position specifying image for specifying a position on the projection image, which is projected by the projector that projects the projection image;
A position specifying unit that associates a position on the projection image with a position on the captured image based on the captured image of the position specifying image;
An operation detection unit for detecting an operation on the captured image;
An instruction information conversion unit that converts the operation content detected by the operation detection unit into instruction information represented by a position on the projection image in accordance with the association by the position specifying unit;
A transmission control unit for causing the projector to transmit the instruction information to the projector;
A portable device comprising: On the computer,
An imaging procedure for causing the imaging unit to image a position specifying image for specifying a position on the projection image, which is projected by the projector that projects the projection image,
An association procedure for associating a position on the projected image with a position on the captured image based on the captured image of the position specifying image;
A detection procedure for detecting an operation on the captured image;
A generation procedure for generating instruction information represented by a position on the projection image from the operation content detected by the detection procedure according to the association by the association procedure;
A transmission procedure for causing the projector to transmit the instruction information to the projector;
A program for running An imaging step for causing the imaging unit to image a position specifying image for specifying a position on the projection image, which is projected by the projector that projects the projection image;
An association step of associating a position on the projected image with a position on the captured image based on the captured image of the position specifying image;
A detection step of detecting an operation on the captured image;
A generation step for generating instruction information represented by a position on the projection image from the operation content detected by the detection step according to the association by the association step.
A transmission step of causing the projector to transmit the instruction information by a transmission device;
A method for controlling a portable device, comprising:

Images