JP2023184001A - Method for controlling control device, control device and display system - Google Patents

Abstract

To provide a method for controlling a control device in which the operation of an operator is detected even if the configuration is simple.SOLUTION: A third control part 350 of a control device 300 executes the steps of: displaying a first image based on the image data on the projection surface 10 by a projector 200; acquiring the captured image obtained by imaging an operator; detecting a physical feature point of the operator based on the captured image; setting a detection region 500 for detecting the operation of the operator in the captured image and then, setting the size of the detection region 500 based on a feature point; detecting a first indication position indicated by the operator from within the detection region 500 of the captured image; acquiring a second indication position which is a position of a first image corresponding to the first indication position; generating a second image based on the second indication position; and displaying the second image by the projector 200.SELECTED DRAWING: Figure 4

Description

The present invention relates to a control method for a control device, a control device, and a display system.

2. Description of the Related Art Conventionally, a technique is known that detects a user's gesture from an image captured by a camera and executes processing corresponding to the detected gesture.
For example, the display device disclosed in Patent Document 1 includes two cameras that photograph an area including the user, and detects the degree of penetration of the user's fingers into a set virtual space based on images taken by these two cameras. . Furthermore, the display device determines the user's operation based on the detected degree of penetration of the finger, and controls the operation of the display device based on the determined operation.

International Publication No. 2018/142524

However, since the display device of Patent Document 1 uses two cameras to detect three-dimensional position coordinates indicating the degree of penetration of the user's fingers, the system configuration becomes complicated.

A control method for a control device according to the present disclosure includes displaying a first image on a display device, acquiring a captured image of an operator, and determining physical characteristic points of the operator based on the captured image. detecting a group of feature points, setting a detection area for detecting an operation of the operator in the captured image, the size of which is based on the group of feature points, and detecting the operation of the operator from within the detection area of the captured image detecting a first indicated position indicated by the first indicated position; obtaining a second indicated position that is a position in the first image corresponding to the first indicated position; and detecting a second indicated position in the second image based on the second indicated position. A control method for a control device includes: generating a second image; and displaying the second image on the display device.

The control device of the present disclosure displays a first image on a display device, acquires a captured image of an operator, and detects a group of physical feature points of the operator based on the captured image. setting a detection area for detecting an operation by the operator in the captured image, the size of which is based on the feature point group; detecting a first indicated position corresponding to the first indicated position; obtaining a second indicated position that is a position in the first image corresponding to the first indicated position; and generating a second image based on the second indicated position. and displaying the second image on the display device.

The display system of the present disclosure includes: a display device that displays a first image; an imaging device that captures an image of an operator; a captured image that captures the operator from the imaging device; and based on the captured image, detecting a group of physical feature points of the operator; setting a detection area in the captured image for detecting an operation of the operator having a size based on the group of feature points; and detecting a first designated position designated by the operator from within the detection area; and obtaining a second designated position that is a position in the first image corresponding to the first designated position; The display device is a display system that includes: a control device that executes the following: generating a second image based on a second designated position; and the display device executes the following operations: displaying the second image.

FIG. 1 is a diagram showing the system configuration of a display system. FIG. 2 is a block diagram showing the configuration of a camera. FIG. 2 is a block diagram showing the configuration of a projector. FIG. 2 is a block diagram showing the configuration of a control device. Diagram showing a wake-up gesture. Diagram showing a sleep gesture. A diagram showing a detection area. Flowchart showing the operation of the control device. Flowchart showing the operation of the control device.

1. Configuration of Display System FIG. 1 is a diagram showing the system configuration of a display system 1. As shown in FIG.
The display system 1 includes a camera 100 that images a user 30, a projector 200 that displays an image on a projection surface 10, and a control device 300 that controls the camera 100 and the projector 200. The control device 300 also operates as an image supply device that supplies images to the projector 200.

FIG. 1 shows an example in which a camera 100 and a control device 300 are connected wirelessly, and a projector 200 and a control device 300 are connected by a wire. The connection between camera 100 and control device 300 may be wired, and the connection between projector 200 and control device 300 may be wireless.

When the camera 100 receives an instruction from the control device 300 or an operation from the user 30, the camera 100 captures a preset range and generates a captured image. For example, the camera 100 is installed so that the upper body of the user 30 is included within the angle of view. Camera 100 transmits the generated captured image to control device 300.

Projector 200 generates image light based on image data supplied from control device 300. The projector 200 projects the generated image light onto the projection surface 10. As a result, an image based on the image data is displayed on the projection surface 10.

The control device 300 is, for example, a notebook, desktop, or tablet personal computer. A smartphone can also be used as the control device 300.

The control device 300 receives an image captured by the camera 100, analyzes the received image, and detects a gesture performed by the user 30. The control device 300 executes a process associated with the detected gesture, and transmits image data indicating the result of the executed process to the projector 200. The projector 200 displays an image on the projection surface 10 based on the image data received from the control device 300. As a result, an image resulting from execution of the process associated with the gesture performed by the user 30 is displayed on the projection surface 10.

2. Camera Configuration FIG. 2 is a block diagram showing the configuration of the camera 100.
The configuration of the camera 100 will be described with reference to FIG. 2.
The camera 100 includes a first wireless interface 110, an imaging section 130, and a first control section 150. Hereinafter, the interface will also be abbreviated as I/F.

The first wireless I/F 110 is, for example, a wireless communication device including a communication circuit compatible with standards such as Bluetooth and Wi-Fi. Bluetooth and Wi-Fi are registered trademarks. The first wireless I/F 110 is wirelessly connected to the control device 300 and performs data communication with the control device 300. Alternatively, instead of the first wireless I/F 110, the camera 100 is provided with a wired I/F compatible with standards such as USB (Universal Serial Bus), and the camera 100 and the control device 300 are connected by wire. There may be.

The imaging unit 130 includes a lens group, an image sensor, a signal processing circuit, and the like. Illustrations of the lens group, image sensor, and signal processing circuit are omitted.
The lens group includes various lenses such as an imaging lens and a focus lens.
The image sensor includes a CMOS (Complementary Metal Oxide Semiconductor) sensor, a CCD (Charge Coupled Device) sensor, and the like. The image sensor photoelectrically converts incident light from an object to generate image signals of three primary colors: red, green, and blue. The image signal is input to a signal processing circuit.
The signal processing circuit performs signal processing such as gain adjustment, A/D conversion, noise removal, color correction, and gamma correction on the image signal input from the image sensor. The signal processing circuit performs signal processing on the image signal to generate a captured image. The signal processing circuit outputs the generated captured image to the first control unit 150.

The first control unit 150 is a computer device including a first storage unit 160 and a first processor 170.

The first storage unit 160 includes a nonvolatile memory such as a ROM (Read Only Memory) and a volatile memory such as a RAM (Random Access Memory). The ROM stores a control program 165 that controls the operation of the camera 100 and various setting information. The RAM is used as a calculation area for the first processor 170 and is used for temporary storage of various data.

The first processor 170 is an arithmetic processing device including one or more processors such as a CPU (Central Processing Unit) and an MPU (Micro-Processing Unit). The first processor 170 executes the control program 165 to control the operation of each part of the camera 100.
The first processor 170 controls the operations of the first wireless I/F 110 and the imaging unit 130. The first processor 170 controls the imaging unit 130 based on an instruction received from the control device 300 or an operation by the user 30, and causes the imaging unit 130 to generate a captured image. The first processor 170 transmits the captured image input from the imaging unit 130 to the control device 300.

3. Configuration of Projector FIG. 3 is a block diagram showing the configuration of the projector 200.
The configuration of the projector 200 will be described with reference to FIG. 3.
The projector 200 includes a first wired I/F 210, an image processing section 220, a frame memory 225, an image projection section 230, and a second control section 250.

The first wired I/F 210 includes, for example, a connection terminal compatible with standards such as HDMI (High-Definition Multimedia Interface) and USB (Universal Serial Bus), and an I/F circuit. HDMI is a registered trademark. The first wired I/F 210 is connected to the control device 300 by wire and receives image data supplied from the control device 300. The first wired I/F 210 outputs the received image data to the image processing section 220.

A frame memory 225 is connected to the image processing section 220 . The image processing unit 220 develops image data input from the second control unit 250 and the first wired I/F 210 into the frame memory 225. The frame memory 225 is configured by, for example, SDRAM (Synchronous Dynamic Random Access Memory).

The image processing unit 220 performs image processing on the image data developed in the frame memory 225, such as resolution conversion processing or resizing processing, correction of distortion aberration, shape correction processing, digital zoom processing, and adjustment of image hue and brightness. I do. The image processing section 220 executes image processing specified by the second control section 250, and performs the processing using parameters input from the second control section 250 as necessary. Furthermore, it is of course possible for the image processing unit 220 to perform a combination of a plurality of image processes among the above. The image processing unit 220 reads the processed image data from the frame memory 225 and outputs the read image data to the image projection unit 230.

The image processing unit 220 and the frame memory 225 are configured by, for example, an integrated circuit. Integrated circuits include LSI (Large Scale Integrated Circuit), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), and FPGA ( Field-Programmable Gate Array), SoC (System-on-a-chip), etc. It will be done. Further, an analog circuit may be included as part of the integrated circuit configuration, or a configuration in which the second control section 250 and the integrated circuit are combined may be used.

The image projection section 230 includes a light source 231, a light modulation device 233, and an optical unit 237.

The light source 231 includes a discharge type light source lamp such as an ultra-high pressure mercury lamp or a metal halide lamp, or a solid state light source such as a light emitting diode or a semiconductor laser. The light emitted by the light source 231 is incident on the light modulation device 233.

The light modulation device 233 includes a transmissive liquid crystal panel 235 in which liquid crystal is sealed between a pair of transparent substrates, as a light modulation element that modulates the light emitted by the light source 231. The liquid crystal panel 235 includes a plurality of pixels arranged in a matrix. The light modulator 233 applies a driving voltage corresponding to the input image data to each pixel of the liquid crystal panel 235, and changes the light transmittance of each pixel to a transmittance corresponding to the image data. When the light emitted from the light source 231 passes through the liquid crystal panel 235, the light emitted from the light source 231 is modulated, and image light corresponding to image data is generated.

The light modulation element included in the light modulation device 233 is not limited to a transmissive liquid crystal panel, and may be, for example, a reflective liquid crystal panel or a DMD (Digital Micromirror Device).

The optical unit 237 includes a projection lens (not shown) and the like, and enlarges and projects the image light modulated by the light modulation device 233 onto the projection surface 10 . As a result, an image corresponding to the image light is displayed on the projection surface 10.

The second control unit 250 is a computer device including a second storage unit 260 and a second processor 270.

The second storage unit 260 includes a nonvolatile memory such as a ROM, and a volatile memory such as a RAM. The ROM stores a control program 265 used to control the operation of the projector 200 and various setting information. The RAM is used as a calculation area for the second processor 270 and is used for temporary storage of various data.

The second processor 270 is an arithmetic processing device including one or more CPUs, processors such as MPUs, and the like. The second processor 270 executes the control program 265 to control the operation of each part of the projector 200.

4. Configuration of Control Device FIG. 4 is a block diagram showing the configuration of the control device 300.
The configuration of the control device 300 will be explained with reference to FIG. 4.
The control device 300 includes a second wireless I/F 310 , a second wired I/F 320 , an operation section 330 , a display section 340 , and a third control section 350 .

The second wireless I/F 310 is, for example, a wireless communication device including a communication circuit compatible with standards such as Bluetooth and Wi-Fi. The second wireless I/F 310 is wirelessly connected to the camera 100 and performs data communication with the camera 100.

The second wired I/F 320 includes, for example, a connection terminal compatible with standards such as HDMI and USB, and an I/F circuit. The second wired I/F 320 is connected to the projector 200 by wire and transmits image data to the projector 200.

The operation unit 330 includes, for example, an input device such as a mouse and a keyboard, and receives operations from the user 30. The operation unit 330 outputs an operation signal corresponding to the received operation to the third control unit 350.

The touch panel 340 includes a display panel 341 such as a liquid crystal panel or an organic EL (Electro Luminescence), and a touch sensor 343. The touch sensor 343 detects a touch operation and outputs an operation signal including the operation position of the detected touch operation to the third control unit 350.

The third control unit 350 is a computer device including a third storage unit 360 and a third processor 370.

The third storage unit 360 includes a nonvolatile memory such as a ROM, and a volatile memory such as a RAM. Further, the third storage unit 360 may be configured to include an auxiliary storage device such as a HDD (Hard Disk Drive) or an SSD (Solid State Drive). The ROM stores an OS (Operating System) 361 that controls the operation of the control device 300, an application program 363, a learning model 365, and calibration data 367. Hereinafter, the application program will be referred to as APP. RAM is used for temporary storage of various data and the like. Further, captured images received from the camera 100 are temporarily stored in the RAM.

The learning model 365 is a machine learning model that detects feature points of a person by machine learning using teacher data, that is, supervised learning. Machine learning models can be constructed using algorithms and data structures. The machine learning method is not limited at all, and various methods such as neural networks and deep learning may be used. The training data is data composed of many combinations of input vectors input to the machine learning model and correct label information to be obtained from the input vectors. In this embodiment, the input vector corresponds to a captured image, and the label information is a feature point of a person. Feature points include joints such as a person's palms, arms, legs, and waist, shoulders, pupils, outer and inner corners of the eyes, and mouth.

The calibration data 367 is data that associates the coordinates of a detection area 500, which will be described later, with the coordinates of a display image displayed on the touch panel 340. That is, the calibration data 367 is conversion data that converts the coordinates of the detection area 500 into the coordinates of the display image. The calibration data 367 is set based on the number of pixels of the display image and the number of pixels of the detection area 500.

The third processor 370 is an arithmetic processing device including one or more CPUs, processors such as MPUs, and the like. The third processor 370 executes the OS 361 and the APP 363 to control the operations of each part of the projector 200. Further, the third processor 370 uses the learning model 365 to detect gestures performed by the user 30. The third processor 370 changes the operation mode of the control device 300 or executes processing corresponding to the detected gesture based on the gesture detection result.

5. Operation of Control Device Next, the operation of the control device 300 will be explained.
The third control unit 350 causes the camera 100 to start capturing an image, and upon receiving a captured image from the camera 100, causes the third storage unit 360 to temporarily store the received captured image.

Next, the third control unit 350 acquires the captured image from the third storage unit 360.
The third control unit 350 uses the learning model 365 to detect feature points such as joints included in the palm or arm, shoulders, pupils, outer corners of the eyes, inner corners of the eyes, mouth, nose, etc. included in the captured image.

Next, the third control unit 350 detects a wake-up gesture, a sleep gesture, and an operation gesture based on the detected feature points.
The wake-up gesture is a gesture performed when changing the operation mode of the control device 300 from the device operation mode to the gesture operation mode.
The sleep gesture is a gesture performed when changing the operation mode of the control device 300 from the gesture operation mode to the device operation mode.

The control device 300 has a device operation mode and a gesture operation mode as operation modes.
The device operation mode is a mode in which operations by the user 30 are accepted using the operation unit 330 or the touch panel 340. The third control unit 350 can, for example, start or terminate the APP 363, open or close a file, select an operation menu included in the APP 363, etc. based on an operation signal input from the operation unit 330 or the touch panel 340 that has received the operation. Execute the process.

The gesture operation mode is a mode in which operations by gestures from the user 30 are accepted.
When the operation mode of the control device 300 is the gesture operation mode, the third control unit 350 analyzes the captured image to detect an operation gesture, and executes processing corresponding to the detected operation gesture.
The operation gesture is a gesture associated with an operation that the third control unit 350 can accept. Operations that can be accepted by the third control unit 350 include operations such as starting and terminating the APP 363 described above, opening and closing a file, and selecting an operation menu included in the APP 363.

FIG. 5 is a diagram illustrating an example of a wake-up gesture.
The wake-up gesture of this embodiment is an action of moving the hand from a position lower than the user's 30 shoulder to a position higher than the shoulder with the palm facing the camera 100 so that the image is captured by the camera 100 and the index finger raised. It is.
The wake-up gesture may be performed with multiple fingers, such as the index finger and middle finger, held up. Further, the wake-up gesture may be performed with a finger other than the index finger, such as the thumb or little finger, held up. Upright state means a state in which the joints of the fingers are extended without being bent.

FIG. 6 is a diagram illustrating an example of a sleep gesture.
In the sleep gesture of this embodiment, the hand is raised higher than the shoulder of the user 30, with the palm facing the camera 100 so that the palm is imaged by the camera 100, and the index finger raised. This is the movement of moving the body from one position to a position lower than the shoulder level.
The sleep gesture may also be performed with multiple fingers such as the index finger and middle finger held up, or may be performed with any finger other than the index finger held up, such as the thumb or little finger.

When the operation mode of the control device 300 is the device operation mode, the third control unit 350 causes the camera 100 to perform imaging, analyzes the captured image received from the camera 100 using the learning model 365, and performs wake-up. Detect gestures.
Further, when the operation mode of the control device 300 is the gesture operation mode, the third control unit 350 analyzes the captured image received from the camera 100 using the learning model 365, and analyzes the operation gesture, instruction operation, and sleep mode. Detect gestures. Details of the instruction operation will be described later.

When the third control unit 350 detects a wake-up gesture in the device operation mode, it sets a detection area 500 within the captured image. The detection area 500 is an area in which an operation gesture is detected in the captured image. By setting the detection area 500, the processing load can be reduced and the accuracy of detecting the operation gesture can be increased compared to the case where the operation gesture is detected from the entire captured image.

First, the third control unit 350 uses the learning model 365 to detect a feature point corresponding to the base of the index finger of the user's 30 palm from the captured image. The feature point detected from the palm of the user 30 corresponds to the second feature point.
The third control unit 350 sets the detected feature point or the center of the feature point to the center of the detection area 500 in the vertical and horizontal directions. The vertical direction is a direction parallel to the vertical side of the captured image, and the horizontal direction is a direction parallel to the horizontal side of the captured image.

Further, the third control unit 350 detects feature points of the user's left and right eyes detected based on the learning model 365. The feature points of the left and right eyes may be, for example, the pupils of the left and right eyes, or the inner and outer corners of the eyes. The feature points of the left and right eyes correspond to the first feature points.
When the third control unit 350 detects the feature points of the left and right eyes, it determines the distance L1 between the detected feature points of the left eye and the right eye, or the distance L2 between the centers of the feature points of the left eye and the right eye. seek. The third control unit 350 calculates the number of pixels of the captured image corresponding to the distance L1 between the feature points of the left eye and the right eye, or the distance L2 between the centers of the feature points of the left eye and the right eye. The distance determined by the identified feature points between the left eye and the right eye will hereinafter be referred to as the distance L between the feature points. As the distance L between the feature points, either the distance L1 between the feature points of the left eye and the right eye, or the distance L2 between the centers of the feature points of the left eye and the right eye can be used.

Next, the third control unit 350 sets the horizontal and vertical widths of the rectangular detection area 500 based on the determined distance L between the feature points. The third control unit 350 calculates the width of the detection area 500 by multiplying the distance L between the feature points obtained by a preset setting value. That is, the lengths of the upper and lower sides, which are two opposing sides of the rectangular detection area 500, are set. The set value can be changed within a range of, for example, 3 or more and 10 or less. Further, this setting value can be changed by the user 30 operating the operation unit 330. For example, if the user 30 wants to set the detection area 500 large, the user 30 sets the setting value to a large value in the range of 3 or more and 10 or less, and if the user 30 wants to set the detection area 500 small, the setting value is set to 3 or more. , set to a small value within the range of 10 or less.

FIG. 7 is a diagram showing a detection area 500 set in a captured image.
The third control unit 350 sets the detection area 500 in the captured image so that the center is located at the feature point corresponding to the base of the index finger or the center of the feature point, and the width is the calculated width. Further, the third control unit 350 sets the vertical width of the detection area 500 based on the calculated horizontal width and the aspect ratio of the display image displayed on the touch panel 340. Specifically, the third control unit 350 sets the vertical width of the detection area 500 so that the aspect ratio of the detection area 500 is the same as the aspect ratio of the display image. The vertical width of the detection area 500 is set by the number of pixels of the captured image.

After setting the detection area 500, the third control unit 350 generates calibration data 367. Calibration data 367 is generated based on the number of pixels in the vertical and horizontal widths of the detection area 500 and the number of pixels in the vertical and horizontal widths of the display image on the touch panel 340.
The calibration data 367 associates the designated position of the user 30 detected from the detection area 500 with the designated position in the image data.

The calibration data 367 is the ratio of the number of pixels in the vertical width of the display image of the touch panel 340 to the number of pixels in the vertical width of the detection area 500, and the number of pixels in the horizontal width of the display image on the touch panel 340 to the number of pixels in the horizontal width of the detection area 500. including the ratio of In this case, by multiplying the coordinate values of the designated position in the detection area 500 by the ratio of the number of pixels described above, the coordinate value of the designated position in the display image of the touch panel 340 can be calculated. In addition, the calibration data 367 is not limited to the ratio of the number of pixels, but is, for example, a conversion formula for projectively converting the coordinates of the four corner points of the detection area 500 in the captured image to the coordinates of the four corner points of the display image on the touch panel 340. It may also be used as calibration data 367.

After setting the detection area 500, the third control unit 350 detects the inside of the set detection area 500 and detects an instruction operation or an operation gesture.
The instruction operation is a process of detecting a position within the detection area 500 instructed by the user 30 as a first instruction position, and determining the position of the display image corresponding to the detected first instruction position using the calibration data 367. The position of the display image converted by the calibration data 367 is referred to as a second designated position. The third control unit 350 determines the process to be executed based on the second designated position of the display image, and executes the determined process.
For example, the third control unit 350 executes a process of moving the mouse cursor to a position corresponding to the second designated position on the display image.

When the third control unit 350 detects an operation gesture, it executes processing corresponding to the detected operation gesture. For example, when an operation gesture associated with a left click is detected while the mouse cursor is displayed overlapping an icon, the APP 363 corresponding to this icon is executed.
The third control unit 350 executes the selected APP 363 and causes the APP screen to be displayed on the touch panel 340. Further, the third control unit 350 transmits image data of the APP screen to the projector 200. The projector 200 generates image light based on the image data received from the control device 300 and projects the generated image light onto the projection surface 10. As a result, the APP screen is displayed on the projection surface 10.

Furthermore, when the third control unit 350 detects a sleep gesture from the captured image, the third control unit 350 changes the operation mode of the control device 300 from the gesture operation mode to the device operation mode.

FIG. 8 is a flowchart showing the operation of the control device 300.
The operation of the control device 300 will be described with reference to the flowchart shown in FIG.
When the projector 200 is connected to the control device 300 via a cable and the power of the projector 200 is turned on, the third control unit 350 transmits image data to the projector 200 (step S1). This image data is, for example, image data that is the source of a display image displayed on the touch panel 340. The projector 200 displays an image on the projection surface 10 based on the image data received from the control device 300. As a result, the display image displayed on the touch panel 340 is displayed on the projection surface 10. The display image displayed on the projection surface 10 corresponds to the first image.

Next, the third control unit 350 instructs the camera 100 to take an image (step S2). The camera 100 starts capturing images when instructed to capture images by the control device 300, and generates captured images at predetermined imaging intervals. Camera 100 transmits the generated captured image to control device 300. The third control unit 350 temporarily stores the captured image received from the camera 100 in the third storage unit 360.

Next, the third control unit 350 acquires the captured image from the third storage unit 360 (step S3), and detects feature points from the captured image using the learning model 365 (step S4).
The third control unit 350 determines whether a wake-up gesture has been detected based on the detection result of the feature points (step S5). The third control unit 350 identifies the body part of the user 30 to which the detected feature point belongs, and determines whether a wake-up gesture is detected.

The third control unit 350 determines that a wake-up gesture has been detected when the following three conditions are met.
The first condition is that feature points included in the palm of the hand are detected.
The second condition is that the feature point of the joint of the index finger is detected and it is determined that the index finger is in an upright position.
The third condition is that the feature point included in the palm has moved from a position lower than the feature point indicating the shoulder position to a position higher than the feature point indicating the shoulder position.

If the third control unit 350 does not detect a wake-up gesture (step S5/NO), the third control unit 350 determines whether the operation unit 330 has accepted the operation (step S6). That is, the third control unit 350 determines whether an operation signal is input from an input device such as a mouse or a keyboard. If the operation unit 330 does not accept the operation (step S6/NO), the third control unit 350 returns to the process of step S3.

When the operation unit 330 receives the operation (step S6/YES), the third control unit 350 determines whether the received operation is a termination operation for terminating the image display by the projector 200 (step S7). ). If the received operation is an end operation (step S7/YES), the third control unit 350 ends this processing flow.
If the received operation is not an end operation (step S7/NO), the third control unit 350 executes a process corresponding to the received operation (step S8), and returns to the process of step S3.

When the third control unit 350 detects a wake-up gesture (step S5/YES), the third control unit 350 determines whether the detection area 500 can be set based on the detected feature points (step S9). The third control unit 350 determines that the detection area 500 cannot be set, for example, if the detection area 500 cannot be set due to a reason such as not being able to detect either one of the feature points of the left or right eyes. If the detection area 500 cannot be set (step S9/NO), the third control unit 350 acquires the captured image again (step S10). The third control unit 350 detects feature points from the acquired captured image using the learning model 365 (step S11), and determines whether the detection area 500 can be set again (step S9).

If the third control unit 350 determines that the detection area 500 can be set (step S9/YES), the third control unit 350 sets the detection area 500 within the captured image (step S12). Further, in step S12, calibration data 367 is generated based on the set detection area 500 and the display image of the touch panel 340.
The third control unit 350 specifies the feature points of the left and right eyes of the user 30 and calculates the distance L between the feature points of the left and right eyes. The third control unit 350 calculates the width of the detection area 500 by multiplying the calculated distance by the set value. Further, the third control unit 350 sets the detection area 500 in the captured image so that the center is located at the feature point corresponding to the base of the index finger or the center of the feature point, and the width is the calculated width. Further, the third control unit 350 sets the vertical width of the detection area 500 based on the calculated horizontal width and the aspect ratio of the image data that the control device 300 supplies to the projector 200. Specifically, the third control unit 350 sets the vertical width of the detection area 500 so that the aspect ratio of the detection area 500 is the same as the aspect ratio of the image data.

Next, the third control unit 350 changes the operation mode of the control device 300 from the device operation mode to the gesture operation mode (step S13), and acquires the next captured image from the third storage unit 360 (step S14). . The third control unit 350 detects feature points from the captured image using the learning model 365 (step S15), and determines whether an instruction operation or an operation gesture has been detected based on the detection result of the feature points (step S15). S16).

When the third control unit 350 detects an instruction operation or an operation gesture (step S16/YES), the third control unit 350 determines whether or not the detected operation gesture is an end operation that ends the display of the image by the projector 200 (step S16/YES). S17). If the detected operation gesture is an end operation (step S17/YES), the third control unit 350 stops transmitting the image data to the projector 200, and ends this processing flow.

Further, if the detected operation gesture is not an end operation (step S17/NO), the third control unit 350 executes a process corresponding to the detected instruction operation or operation gesture (step S18). If an instruction operation is detected, the third control unit 350 calculates the instruction position in the display image of the touch panel 340 based on the calibration data 367 in step S18, and executes processing according to the calculated instruction position. do.
Further, the third control unit 350 generates image data corresponding to the execution result of the process (step S19). The third control unit 350 transmits the generated image data to the projector 200 (step S20). By displaying an image based on the image data received by the projector 200 on the projection surface 10, the second image, which is the display image displayed on the touch panel 340, is displayed on the projection surface 10.

Further, if the instruction operation or operation gesture is not detected in step S16 (step S16/NO), the third control unit 350 determines whether a sleep gesture has been detected (step S21). The third control unit 350 determines that a sleep gesture has been detected when the following three conditions are met.
The first condition is that feature points included in the palm of the hand are detected.
The second condition is that the feature point of the joint of the index finger is detected and it is determined that the index finger is in an upright position.
The third condition is that the feature point included in the palm has moved from a position higher than the feature point indicating the shoulder position to a position lower than the feature point indicating the shoulder position.

If the third control unit 350 does not detect a sleep gesture (step S21/NO), the process returns to step S14.
Further, when the third control unit 350 detects a sleep gesture (step S21), the third control unit 350 changes the operation mode of the control device 300 from the gesture operation mode to the device operation mode (step S22), and returns to the process of step S3.

6. Summary of the present disclosure A summary of the present disclosure is appended below.
(Additional note 1)
Displaying the first image on a display device;
Obtaining a captured image of the operator;
Detecting a group of physical feature points of the operator based on the captured image;
setting a detection area in the captured image for detecting the operation of the operator, the size of which is based on the feature point group;
detecting a first designated position designated by the operator from within the detection area of the captured image;
obtaining a second designated position that is a position in the first image corresponding to the first designated position;
generating a second image based on the second designated position;
Displaying the second image on the display device;
A control method for a control device including:

Thereby, the size of the detection area set in the captured image is set based on the physical feature points of the operator detected from the captured image. Therefore, by detecting the first designated position from the detection area, the operator's operation can be detected even with a simple configuration.

(Additional note 2)
Detecting the group of physical feature points of the operator includes:
detecting the position of the left eye and the position of the right eye of the operator as a first feature point group included in the feature point group;
including;
Setting the detection area in the captured image includes:
The display method according to supplementary note 1, including setting the size of the detection area in the captured image based on the distance from the position of the operator's left eye to the position of the right eye in the captured image.

Thereby, the size of the detection area is set based on the distance between the left and right eyes of the operator. Therefore, the size of the detection area in the captured image can be set to a size suitable for the operator's operation, regardless of the imaging position of the captured image and the distance from the operator.

(Additional note 3)
Detecting the group of physical feature points of the operator includes:
detecting the position of the left shoulder and the position of the right shoulder of the operator as a first feature point group included in the feature point group;
including;
Setting the detection area in the captured image includes:
The display method according to supplementary note 1, including setting the size of the detection area in the captured image based on the distance from the left shoulder position to the right shoulder position of the operator in the captured image.

Thereby, the size of the detection area is set based on the distance between the left and right shoulders of the operator. Therefore, the size of the detection area in the captured image can be set to a size suitable for the operator's operation, regardless of the imaging position of the captured image and the distance from the operator.

(Additional note 4)
Detecting the group of physical feature points of the operator includes:
Detecting a feature point included in the operator's hand as a second feature point included in the feature point group;
including;
Setting the detection area in the captured image includes:
setting the detection area based on the position of the second feature point;
The display method according to any one of Supplementary Notes 1 to 3, including:

Thereby, the detection area is set based on the position of the feature point included in the operator's hand. Therefore, the detection area can be set in a range that includes the user's hand.

(Appendix 5)
Detecting the group of physical feature points of the operator includes:
Detecting a feature point included in the operator's hand as a second feature point included in the feature point group;
including;
Setting the detection area in the captured image includes:
setting the detection area in the captured image based on the group of feature points when a preset operation is detected based on the second feature point;
The display method according to any one of Supplementary Notes 1 to 3, including:

Thereby, when a preset operation is detected based on the second feature point included in the operator's hand, the detection area is set in the captured image.
Therefore, the detection area can be set in the captured image by the operator performing a preset operation.

(Appendix 6)
Detecting the group of physical feature points of the operator includes:
Detecting a joint position of the index finger of the operator as a second feature point included in the feature point group;
including;
Setting the detection area in the captured image includes:
setting the detection area based on the joint position of the index finger;
The display method according to any one of Supplementary Notes 1 to 3, including:

Thereby, the detection area is set in the captured image based on the joint position of the operator's index finger. Therefore, the detection accuracy of the operation performed by the operator in the detection area can be improved.

(Appendix 7)
Setting the detection area in the captured image includes:
The detection area is set as a rectangle whose length on two opposing sides is a value obtained by multiplying the distance from the position of the operator's left eye to the position of the right eye in the captured image by a preset setting value. to do,
The display method described in Appendix 2, including.

Thereby, the size of the detection area is set based on the distance between the left and right eyes of the operator. Therefore, the size of the detection area can be set to a size suitable for the operator's operations.

(Appendix 8)
Displaying the first image on a display device;
Obtaining a captured image of the operator;
Detecting a group of physical feature points of the operator based on the captured image;
setting a detection area in the captured image for detecting the operation of the operator, the size of which is based on the feature point group;
detecting a first designated position designated by the operator from within the detection area of the captured image;
obtaining a second designated position that is a position in the first image corresponding to the first designated position;
generating a second image based on the second designated position;
Displaying the second image on the display device;
A control device that executes.

Thereby, the size of the detection area set in the captured image is set based on the physical feature points of the operator detected from the captured image. Therefore, by detecting the first designated position from the detection area, the detection accuracy of the operator's operation can be improved even with a simple configuration.

(Appendix 9)
a display device that displays a first image;
an imaging device that captures an image of an operator;
Obtaining a captured image of the operator from the imaging device;
Detecting a group of physical feature points of the operator based on the captured image;
setting a detection area in the captured image for detecting the operation of the operator, the size of which is based on the feature point group;
detecting a first designated position designated by the operator from within the detection area of the captured image;
obtaining a second designated position that is a position in the first image corresponding to the first designated position;
generating a second image based on the second designated position;
a control device that performs
Equipped with
The display device includes:
Displaying the second image;
A display system that runs.

Thereby, the size of the detection area set in the captured image is set based on the physical feature points of the operator detected from the captured image. Therefore, by detecting the first designated position from the detection area, the detection accuracy of the operator's operation can be improved even with a simple configuration.

The embodiments described above are preferred embodiments of the present invention. However, the present invention is not limited to this embodiment, and various modifications can be made without departing from the gist of the present invention.
For example, in the embodiment described above, a case has been described in which the control device 300 detects a wake-up gesture, a sleep gesture, and an operation gesture, but the camera 100 or the projector 200 may also detect the gesture. The control device 300 changes the operation mode based on a wake-up gesture or a sleep gesture detected by the camera 100 or the projector 200. Further, the control device 300 executes processing associated with the operation gesture detected by the camera 100 or the projector 200.

Furthermore, in the embodiment described above, a feature point corresponding to the base of the index finger of the user's 30 palm is identified, and the identified feature point is set at the center of the detection area 500 in the vertical and horizontal directions. The feature point set at the center of the detection area 500 in the vertical and horizontal directions may be the tip of a finger such as an index finger.

Furthermore, in the embodiment described above, the left and right eyes of the user 30 are detected as the second feature points, and the width of the detection area 500 is set based on the detected distance between the left and right eyes.
As a modified example, the third control unit 350 detects the left and right shoulder positions of the user 30 as the second feature points, and multiplies the detected left and right shoulder positions by a preset value to determine the area of the detection area 500. Calculate the width.

Further, each functional unit of the camera 100 shown in FIG. 2, the projector 200 shown in FIG. 3, and the control device 300 shown in FIG. 4 shows a functional configuration realized by cooperation between hardware and software. Therefore, the specific implementation form is not particularly limited. Therefore, it is not necessarily necessary to implement hardware corresponding to each functional unit individually, and it is of course possible to have a configuration in which one processor executes a program to realize the functions of a plurality of functional units. Moreover, some of the functions realized by software in the above embodiments may be realized by hardware, and some of the functions realized by hardware may be realized by software.

Further, when the above-described control method for the control device is implemented using a computer installed in the control device 300, it is also possible to configure a program to be executed by the computer in the form of a recording medium. Further, the program executed by the computer can also be configured in the form of a transmission medium that transmits the program. A magnetic or optical recording medium or a semiconductor memory device can be used as the recording medium. Specifically, portable types such as flexible discs, HDDs, CD-ROMs (compact disc read-only memory), DVDs (digital versatile discs), Blu-ray discs, magneto-optical discs, flash memories, and card-type recording media; Alternatively, a fixed recording medium may be used. Further, the recording medium may be a nonvolatile storage device such as a RAM, ROM, or HDD, which is an internal storage device included in the server device. Blu-ray is a registered trademark.

Furthermore, the processing units in the flowcharts shown in FIGS. 8 and 9 are divided according to the main processing contents in order to facilitate understanding of the processing of the third control unit 350. The present invention is not limited by the method of dividing the processing units or the names shown in FIG. Further, the processing of the second control unit 250 can be divided into more processing units depending on the processing content, or can be divided so that one processing unit includes even more processing. Furthermore, the processing order in the above flowchart is not limited to the illustrated example.

Further, in the embodiment described above, a configuration in which the projector 200 is used as the display device is illustrated, but the display device is not limited to the projector 200. For example, a self-luminous display device such as a monitor or a liquid crystal television, such as a liquid crystal display device that displays images on a liquid crystal display panel, or a display device that displays images on an organic EL panel, may be used.

Further, in the embodiment described above, the display system 1 in which the camera 100, the projector 200, and the control device 300 are each separate bodies is illustrated, but the present invention is not limited to this display system 1. For example, the projector 200 may have a built-in camera 100, or the control device 300 may have a built-in camera 100. Furthermore, the projector 200 may have a configuration in which the control device 300 is built in, or the projector 200 may have a configuration in which the control device 300 and the camera 100 are built in.

Here, when the projector 200 includes the control device 300, it may be configured without the touch panel 340. In this case, the vertical width of the detection area 500 may be set based on the calculated horizontal width and the aspect ratio of the display image displayed by the liquid crystal panel 235 included in the projector 200. Further, the calibration data 367 may be generated based on the number of pixels in the vertical and horizontal widths of the detection area 500 and the number of pixels in the vertical and horizontal widths of the display image displayed by the liquid crystal panel 235.

DESCRIPTION OF SYMBOLS 1... Display system, 10... Projection surface, 30... User, 100... Camera, 110... First wireless interface, 130... Imaging section, 150... First control section, 160... First storage section, 165... Control program, 170 ... first processor, 200 ... projector, 210 ... first wired I/F, 220 ... image processing section, 225 ... frame memory, 230 ... image projection section, 231 ... light source, 233 ... light modulation device, 235 ... liquid crystal panel, 237...Optical unit, 250...Second control section, 260...Second storage section, 265...Control program, 270...Second processor, 300...Control device, 310...Second wireless I/F, 320...Second wired I /F, 330...Operation unit, 340...Touch panel, 343...Touch sensor, 350...Third control unit, 360...Third storage unit, 361...OS, 363...APP, 365...Learning model, 367...Calibration data, 370...Third processor, 500...Detection area.

Claims (9)

Displaying the first image on a display device;
Obtaining a captured image of the operator;
Detecting a group of physical feature points of the operator based on the captured image;
setting a detection area in the captured image for detecting the operation of the operator, the size of which is based on the feature point group;
detecting a first designated position designated by the operator from within the detection area of the captured image;
obtaining a second designated position that is a position in the first image corresponding to the first designated position;
generating a second image based on the second designated position;
Displaying the second image on the display device;
A control method for a control device including: Detecting the group of physical feature points of the operator includes:
detecting the position of the left eye and the position of the right eye of the operator as a first feature point group included in the feature point group;
including;
Setting the detection area in the captured image includes:
The control device according to claim 1, further comprising: setting the size of the detection area in the captured image based on a distance from the position of the left eye to the position of the right eye of the operator in the captured image. control method. Detecting the group of physical feature points of the operator includes:
detecting the position of the left shoulder and the position of the right shoulder of the operator as a first feature point group included in the feature point group;
including;
Setting the detection area in the captured image includes:
The control device according to claim 1, further comprising: setting the size of the detection area in the captured image based on a distance from a left shoulder position to a right shoulder position of the operator in the captured image. Control method. Detecting the group of physical feature points of the operator includes:
Detecting a feature point included in the operator's hand as a second feature point included in the feature point group;
including;
Setting the detection area in the captured image includes:
setting the detection area based on the position of the second feature point;
A method for controlling a control device according to any one of claims 1 to 3, comprising: Detecting the group of physical feature points of the operator includes:
Detecting a feature point included in the operator's hand as a second feature point included in the feature point group;
including;
Setting the detection area in the captured image includes:
setting the detection area in the captured image based on the group of feature points when a preset operation is detected based on the second feature point;
A method for controlling a control device according to any one of claims 1 to 3, comprising: Detecting the group of physical feature points of the operator includes:
Detecting a joint position of the index finger of the operator as a second feature point included in the feature point group;
including;
Setting the detection area in the captured image includes:
setting the detection area based on the joint position of the index finger;
A method for controlling a control device according to any one of claims 1 to 3, comprising: Setting the detection area in the captured image includes:
The detection area is set as a rectangle whose length on two opposing sides is a value obtained by multiplying the distance from the position of the operator's left eye to the position of the right eye in the captured image by a preset setting value. to do,
A method for controlling a control device according to claim 2, comprising: Displaying the first image on a display device;
Obtaining a captured image of the operator;
Detecting a group of physical feature points of the operator based on the captured image;
setting a detection area in the captured image for detecting the operation of the operator, the size of which is based on the feature point group;
detecting a first designated position designated by the operator from within the detection area of the captured image;
obtaining a second designated position that is a position in the first image corresponding to the first designated position;
generating a second image based on the second designated position;
Displaying the second image on the display device;
A control device that executes. a display device that displays a first image;
an imaging device that captures an image of an operator;
Obtaining a captured image of the operator from the imaging device;
Detecting a group of physical feature points of the operator based on the captured image;
setting a detection area in the captured image for detecting the operation of the operator, the size of which is based on the feature point group;
detecting a first designated position designated by the operator from within the detection area of the captured image;
obtaining a second designated position that is a position in the first image corresponding to the first designated position;
generating a second image based on the second designated position;
a control device that performs
Equipped with
The display device includes:
Displaying the second image;
execute,
display system.

Images