WO2022034745A1 - Information processing device for superimposing write screen image - Google Patents

Abstract

[Problem] To provide an information processing device, an information processing method, and a program which can increase the convenience of user input by displaying information about the user input in a superimposed manner on screens presented by different applications. [Solution] This information processing device includes a control unit which displays and outputs a first screen presented by a first application, and a second screen with a transmissive background presented by a second application that accepts write input. The control unit performs control to display the second screen on the first screen in a superimposed manner, and performs image processing that uses the first screen and the second screen including the display of the write input.

Description

[Name of the invention determined by ISA based on Rule 37.2.] Information processing device that superimposes the writing screen image

This disclosure relates to information processing devices, information processing methods, and programs.

In recent years, when a user moves a drawing tool on a drawing surface in real space, a drawing system that projects information such as a line indicating the trajectory of the drawing tool (hereinafter, also referred to as drawing information) onto the drawing surface by a projection device such as a projector has been used. It has appeared. For example, in Patent Document 1 below, infrared light emitted from a digital pen, which is a drawing tool used in such a drawing system, is detected to recognize handwriting drawn by the digital pen, and the pen tip of the digital pen touches the ground. A technique for projecting a handwriting image on an existing object (drawing surface) is disclosed.

International Publication No. 2019/11146

In Patent Document 1, the handwriting image input to the real space by the digital pen is saved and read, and the handwriting image is projected (displayed) to the real space, but it is developed by other applications such as existing applications. It is not considered to be superimposed on the displayed image.

According to the present disclosure, a control unit that displays and outputs a first screen presented by a first application and a second screen with a transparent background presented by a second application that accepts write input. The control unit controls to superimpose and display the second screen on the first screen, and displays the second screen including the display of the write input and the first screen. We propose an information processing device that performs the image processing used.

According to the present disclosure, the processor displays and outputs a first screen presented by the first application and a second screen with a transparent background presented by the second application accepting write input. The second screen including the display of the write input and the first screen are used to perform control to superimpose and display the second screen on the first screen. We propose information processing methods including image processing.

According to the present disclosure, a computer is displayed and output as a first screen presented by a first application and a second screen with a transparent background presented by a second application that accepts write input. The control unit controls to superimpose and display the second screen on the first screen, and the second screen including the display of the write input and the first screen. We propose a program that performs image processing using the screen of.

It is a figure explaining the outline of the information processing system by one Embodiment of this disclosure. It is a figure which shows an example of the structure of the information processing system which concerns on this embodiment. It is a figure explaining the functional structure of the writing application by this embodiment. It is a figure which shows an example of the projection image which superposed and displayed the writing screen image by this embodiment. It is a figure explaining the superimposition of two images by this embodiment. It is a flowchart which shows the operation process of the information processing apparatus by this embodiment. It is a sequence diagram which shows an example of the operation processing of the writing display application and the writing application by this embodiment. It is a figure explaining the case of generating the composite image with the image to be written based on the writing screen image by this embodiment. It is a figure which shows an example of the notification of the pen input state and the end icon by this embodiment. It is a flowchart which shows an example of the operation processing of the general application and the writing application by this embodiment. It is a figure explaining two kinds of eraser functions by this embodiment. It is a sequence diagram which shows the operation process when there is the cooperation of the eraser function by this embodiment. It is a flowchart which shows the operation process when there is no cooperation of the eraser function by this embodiment. It is a figure which shows an example of the selection GUI of the eraser function by this embodiment. It is a figure explaining the selection of the eraser function by switching the drawing tool by this embodiment. It is a figure explaining the adjustment of the display position of a bubble image according to the position of a user's head by this embodiment.

The preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings below. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted.

In addition, the explanation shall be given in the following order.
1. 1. Outline of information processing system according to one embodiment of the present disclosure 2. Configuration example 2-1. Display device 200
2-2. Sensor 300
2-3. Digital pen 400
2-4. Information processing device 100
(2-4-1. Functional configuration of writing application)
(2-4-2. Overlapping display of writing screen image)
3. 3. Operation processing 3-1. First operation processing example 3-2. Second operation processing example 3-3. Third operation processing example 4. Eraser function 4-1. When there is an eraser function linkage 4-2. When there is no cooperation of the eraser function 4-3. Switching the eraser function 5. Others 5-1. Adjusting the position of the writing area display 5-2. Change in writable area 6. supplement

<< 1. Outline of information processing system according to one embodiment of the present disclosure >>
FIG. 1 is a diagram illustrating an outline of an information processing system according to an embodiment of the present disclosure. As shown in FIG. 1, the information processing system according to the present embodiment includes a projector 210, a camera 310, a digital pen 400, and an information processing device 100.

The projector 210 is a projection device that projects an image on an arbitrary place in real space. The projector 210 projects an image into the real space included in the projection angle of view. The projected angle of view means a projectable range and is also called a projection area. The projection area 211 is defined by the position of the projector 210, the projection direction, and the angle of the projectable range about the projection direction as the central axis. The image projected by the projector 210 is also referred to as a projected image. It is assumed that the projected image is an image projected on the entire projection area 211, and drawing information corresponding to the operation of the digital pen 400 by the user is mapped and projected in the projected image. In the example shown in FIG. 1, a handwriting image L showing a movement locus on the drawing surface (corresponding to the projection area 211) of the digital pen 400 is projected as drawing information. In the present embodiment, the projector 210 is an example of a display device, the projection area is an example of a display area, and the projected image is an example of a display image.

The camera 310 is an imaging device that captures images in real space. The camera 310 has a lens system, a drive system, and an image pickup element such as an IR (infrared) camera and an RGB (red, green, blue) camera, and captures an image (still image or moving image). The camera 310 images the real space included in the image pickup angle of view 311. The image pickup angle of view 311 means an image pickup range, and is defined by the installation position of the camera 310, the image pickup direction, and the angle of the image pickup range centered on the image pickup direction. The image captured by the camera 310 is also referred to as a captured image. The image pickup angle of view 311 of the camera 310 according to the present embodiment may be a range including at least the projection area 211.

The digital pen 400 is an input device in which a light emitting unit such as an IR (infrared light) LED (Light Emitting Diode) is mounted on the pen tip. The light emitting unit emits light by operating a button or switch provided on the digital pen 400, pressing the pen tip against the ground surface, swinging the pen, or the like. Further, the digital pen 400 may transmit detection information such as a user operation of a button or a switch provided on the digital pen 400 or a movement of the digital pen 400 to the information processing apparatus 100.

The information processing apparatus 100 detects a light emitting point from the captured image captured by the camera 310 and recognizes the position of the digital pen 400. The imaging angle of view 311 is preset in a range including at least the projection region 211. Alternatively, the projection area 211 may be divided and photographed by a plurality of cameras 310. The information processing apparatus 100 continuously recognizes, that is, tracks the position of the digital pen 400 based on the captured image continuously captured by the camera 310. Further, the information processing apparatus 100 wirelessly connects to the digital pen 400, transmits a light emitting command of the digital pen 400, and receives information indicating that the switch operation is performed by the digital pen 400. Further, when the information processing apparatus 100 receives information from the digital pen 400 indicating that the pen tip is pressed against the ground surface (switch ON state), the information processing apparatus 100 displays the movement trajectory of the digital pen 400 as a handwriting on the projected image. Take control. As a result, the user can perform the drawing operation as if he / she was actually writing in the real space with a pen or the like.

(Arrangement of issues)
Here, when performing pen input such as inputting handwritten characters using a digital pen on the touch display, the content displayed in a dedicated application that can acquire pen input information (display image developed by the dedicated application). ) Was only done. Therefore, it is necessary to read the material to be written into the dedicated application as image data or the like in advance. Further, since the content written in the read image data is data integrated with the image data, it is difficult to process only the written content.

Also, if you want to perform pen input with an existing application, you can think of a method of inputting by making the pen input look like a mouse event, but it was difficult to fully utilize the pen input function. In addition, it was difficult to perform pen input to an existing application that does not support pen input.

Therefore, in this disclosure, we propose a mechanism that can enhance the convenience of user input by superimposing user input information on screens presented by different applications.

<< 2. Configuration example >>
FIG. 2 is a diagram showing an example of the configuration of the information processing system according to the present embodiment. As shown in FIG. 2, the information processing system according to the present embodiment includes an information processing device 100, a display device 200, a sensor 300, and a digital pen 400.

<2-1. Display device 200>
The display device 200 displays an image that reflects the drawing information input by the digital pen 400. In this embodiment, as an example, a projector 210 that projects an image into a real space is used as shown in FIG. 1, but a display may also be used. Further, the display device 200 may be realized by a screen and a rear projector that illuminates the screen from the opposite side (back side). Further, the projector may be, for example, a device having a drive mechanism and capable of projecting in any direction. By having such a mechanism, it is possible to display an image not only in one place but also in various places, and it is possible to realize an interactive function such as projecting in the direction in which the user is. Further, a plurality of display devices 200 may be provided. By displaying an image reflecting drawing information on each display device 200, more users can enjoy drawing at the same time. For example, if it is set so that three users can draw each image projected by one projector 210a (when input by three digital pens 400 is possible), another projector 210b May be set so that three more users can draw on the image projected by the projector 210b. By projecting the images projected by the projector 210a and the projector 210b side by side on a wall or the like in a real space, six users can enjoy drawing at the same time.

Further, the display device 200 may include a component capable of output other than the display. For example, the display device 200 may be combined with a sound output device such as a speaker. The speaker may be a unidirectional speaker capable of forming directivity in a single direction. The unidirectional speaker outputs sound in the direction in which the user is, for example. The sound output device may be provided in a place different from the display device 200.

<2-2. Sensor 300>
The sensor 300 detects a user operation on the projection area. In this embodiment, as an example, a camera 310 that captures a projection region is used as shown in FIG. The camera 310 detects the light emission of the digital pen 400 held by the user. The light emission of the digital pen 400 is assumed to be the light emission of the IR LED provided on the pen tip as described later. In this case, in order to avoid malfunction due to light other than IR emission by the digital pen 400, the camera 310 may be configured to take an image through a visible light cut filter (for example, an optical filter that mainly passes light in the IR wavelength band). .. That is, an IR camera may be used as the camera 310. Further, the number of cameras 310 may be plural, and may further include, for example, an RGB camera.

Further, the camera 310 is assumed to be a camera capable of observing the area displayed by the display device 200, and in the present embodiment, it is assumed that the camera 310 is arranged parallel to and in the vicinity of the optical axis direction of the projector 210. The arrangement position of the projector 210 and the camera 310 is not particularly limited, but the rotation component can be suppressed by making the arrangement positions of the projector 210 and the camera 310 as close and parallel as possible. The rotation component is, for example, a conversion amount generated when performing coordinate conversion from projector coordinates to screen coordinates. In this system, a projection transformation matrix between the camera 310 and the projector 210 may be generated in advance, and the projection region may be cut out from the captured image or the projected image may be deformed using the projection transformation matrix.

Further, the sensor 300 may include other sensors that sense various information in addition to the camera 310. For example, the sensor 300 may sense information such as the position of the user and the height of the user in addition to the user's operation on the projection area. For example, the sensor 300 may further include a depth sensor, a microphone, and the like. The depth sensor is a device that acquires depth information such as an infrared range measuring device, an ultrasonic range measuring device, a LiDAR (Laser Imaging Detection and Ringing), or a stereo camera. Further, the depth sensor may be a ToF (Time Of Flight) camera capable of acquiring a highly accurate distance image. A microphone is a device that collects ambient sound and outputs audio data converted into a digital signal via an amplifier and an ADC (Analog Digital Converter). The microphone may be an array microphone. Further, the sensor 300 may sense the user operation with respect to the projection area by a sensor other than the camera 310. Therefore, for example, the sensor 300 may further include a touch sensor or the like provided in the projection area.

<2-3. Digital pen 400>
The digital pen 400 is an example of an input device used by a user. The digital pen 400 is used when the user draws on the projection area 211. Further, the digital pen 400 is formed by a pen-shaped device as an example. As shown in FIG. 2, the digital pen 400 includes a communication module 410, a control unit 420, an IR LED 430, and a switch 440.

The IR LED 450 is an example of a light emitting unit and emits infrared light. The IR LED 450 lights up under the control of the control unit 420. The emission of such infrared light (the bright spot of the IR LED) is detected by the camera 310, and the position can be recognized by the information processing apparatus 100. The IR LED 450 is provided at the tip (pen tip) of the digital pen 400 formed by the pen-shaped device. As a result, the drawing position when the pen tip is grounded to the drawing surface and drawing is performed can be detected by the camera 310. In the present embodiment, a pen-shaped device is used as an example of the input device, but the present disclosure is not limited to this, and if the bright spot of light emission indicating the drawing position can be detected, the device is formed with a different shape. It may have been done. For example, it may be something that the user sprays, such as a spray can, or it may be a device that can be worn on the limbs.

The switch 440 determines whether or not drawing is being performed by the digital pen 400 (that is, an input state). For example, when the pen tip of the digital pen 400 is pressed against a physical object (drawing surface) such as a wall, the pen tip is pushed in and the switch is turned on. It has a mechanism. The control unit 420 transmits the ON / OFF state of the switch to the information processing apparatus 100. Here, as an example, a mechanism in which the switch is turned on by pressing the pen tip against the drawing surface (ground surface) has been described, but the present disclosure is not limited to this, and a button or switch provided on the digital pen 400 or the like is described. The switch may be turned on by the operation detection by the operation detection, the operation detection to the operation body using the motion sensor of the operation body, or the like.

The communication module 410 is connected to the information processing device 100 by wire or wirelessly to transmit and receive data. The communication module 410 is connected to the information processing device 100 by, for example, a wired / wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), ZigBee (registered trademark), short-range wireless communication, or the like. do.

The control unit 420 functions as an arithmetic processing unit and a control device, and controls the overall operation in the digital pen 400 according to various programs. Further, the control unit 420 is realized by an electronic circuit such as a CPU (Central Processing Unit), a microprocessor, or a microcontroller. Further, the control unit 420 may include a ROM (Read Only Memory) for storing programs to be used, calculation parameters, and the like, and a RAM (Random Access Memory) for temporarily storing parameters and the like that change as appropriate.

The control unit 420 according to the present embodiment controls to transmit the ON / OFF state of the switch 440 and the operation state of other switches and buttons (not shown) from the communication module 410 to the information processing device 100. Further, the control unit 420 controls the lighting of the IR LED 450 according to the lighting control command of the IR LED 450 received from the information processing apparatus 100 by the communication module 410.

In the present embodiment, as an example, a case where a drawing input for drawing a locus is performed using the digital pen 400 has been mainly described, but the operation using the digital pen 400 according to the present embodiment is not limited to the drawing input and is displayed. It is also possible to perform various operations such as erasing the displayed locus image and clicking on the displayed image.

Further, the digital pen 400 may be further provided with a color LED. The color LED lights up in any color under the control of the control unit 420. The control unit 420 may control the lighting of the color LED 430 according to the lighting control command of the color LED received from the information processing apparatus 100 by the communication module 410. The color LED is provided, for example, at the tip of the digital pen 400. The color of the color LED may be controlled to be the same as the color of the handwriting of the handwriting image displaying the locus drawn by the digital pen 400. The color of the handwriting is determined and shared by the information processing apparatus 100 or the digital pen 400. By controlling the color of the pen tip and the color of the handwriting in the same way, the entertainment of the drawing experience in the real space is further enhanced. Further, the color LED may be a visible light LED.

<2-4. Information processing device 100>
The information processing apparatus 100 includes an I / F (Interface) unit 110, a control unit 120, an operation input unit 130, a display unit 140, and a storage unit 150. The information processing device 100 is realized by, for example, a smartphone, a tablet terminal, a PC (personal computer), or the like. Further, the information processing apparatus 100 may be arranged in the same space as the projector 210, the camera 310, and the digital pen 400, or may be a server on the Internet. The information processing apparatus 100 may be realized by, for example, an edge server, an intermediate server, a cloud server, or the like.

(I / F section 110)
The I / F unit 110 is a connection device for connecting the information processing device 100 and other devices. The I / F unit 110 includes, for example, a USB (Universal Serial Bus) connector, a wired / wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), ZigBee (registered trademark), and a mobile communication network (registered trademark). It is realized by LTE (Long Term Evolution), 3G (3rd generation mobile communication method), 4G (4th generation mobile communication method), 5G (5th generation mobile communication method), and the like. The I / F unit 110 inputs / outputs information to / from the projector 210 included in the display device 200, the camera 310 included in the sensor 300, and the digital pen 400.

(Control unit 120)
The control unit 120 functions as an arithmetic processing unit and a control device, and controls the overall operation in the information processing device 100 according to various programs. The control unit 120 is realized by an electronic circuit such as a CPU (Central Processing Unit) or a microprocessor. Further, the control unit 120 may include a ROM (Read Only Memory) for storing programs to be used, calculation parameters, and the like, and a RAM (Random Access Memory) for temporarily storing parameters and the like that change as appropriate.

The control unit 120 according to the present embodiment functions as an application execution unit 121, a display image generation unit 122, and a display control unit 123.

-Application execution unit 121
The application execution unit 121 starts an application (application program) and executes various command execution processes according to the application program. The storage unit 150 stores data of one or more applications, and the application execution unit 121 calls a specific application from the storage unit 150 and executes it according to an instruction. In the present embodiment, a case where the first application and the second application different from the first application are executed by the application execution unit 121 will be described.

The second application is, for example, a writing (drawing input) application that outputs the content input (drawn) by the user to the display device 200 on a screen with a transparent background. More specifically, the writing application detects the position (position on the drawing surface in the real space) of the digital pen 400, which is an example of the input device, by the sensor 300, and detects the locus drawn on the drawing surface by the digital pen 400. Generate a handwriting image to display. The writing application may also identify the positions of the plurality of digital pens 400 and generate a plurality of handwriting images. Further, the writing application maps one or more handwriting images and outputs a screen (writing screen image) through which the background is transparent. Further, the second application is software that appropriately performs image processing based on the writing screen image saved after the writing is completed and the data of the screen (written screen image) presented by the first application. You may. For example, the second application may generate an image (composite process) in which the writing screen image is superimposed on the written screen image after the writing is completed (after the writing screen image is saved).

The first application is an application that presents an image (display image) to be displayed on the display device 200. More specifically, the first application is a general application such as a text creation application, a presentation application, a spreadsheet application, an electronic document viewing / editing application generated in a predetermined format, a photo or video viewing application, and the like. Application may be. Further, the first application acquires some processing (image processing) based on the writing screen image, such as acquiring the writing screen image generated and saved by the second application and performing image composition based on the writing screen image. It may be software that can be used.

The application execution unit 121 controls to superimpose the writing screen image (second screen) presented by the second application on the display image (first screen) presented by the first application. I do. Such control may be performed according to the second application or may be performed according to the first application. Alternatively, it may be performed according to the operation input by the administrator who operates the information processing apparatus 100. By superimposing a writing screen image with a transparent background on the display image presented by the first application, it is not necessary to implement a drawing input function by an input device such as a digital pen 400 on the first application side. It is possible to present the writing (drawing input) state to the image presented by the first application. Specific examples will be described later with reference to FIGS. 4 and 5.

-Display image generator 122
The display image generation unit 122 generates a display image (projected image) to be projected by the projector 210. For example, the display image generation unit 122 is in a state in which a writing screen image with a transparent background presented by the second application is superimposed on the display image presented by the first application executed by the application execution unit 121. Generate a display image.

-Display control unit 123
The display control unit 123 controls the display image generated by the display image generation unit 122 to be output to the projector 210 via the I / F unit 110 and projected by the projector 210.

(Operation input unit 130)
The operation input unit 130 has a function of receiving an operation input from the user. The operation input unit 130 may be realized by an input device such as a keyboard, a mouse, a touch panel, a button, or a switch. It may be assumed that the operation input to the information processing apparatus 100 is performed by an administrator who is a user different from one or more users who write on the drawing surface using the digital pen 400. The administrator may operate the first application and the second application, start / end the projection by the projector 210, and the like.

(Display unit 140)
The display unit 140 is a display that displays various operation screens and the like for operating the information processing apparatus 100. The display unit 140 is realized by a display device such as a liquid crystal display (LCD) or an organic EL ((Electro Luminescence) display), and the display unit 140 also has a display image projected by the projector 210. It may be displayed.

(Memory unit 150)
The storage unit 150 is realized by a ROM (Read Only Memory) that stores programs and arithmetic parameters used for processing of the control unit 120, and a RAM (Random Access Memory) that temporarily stores parameters and the like that change as appropriate. For example, the storage unit 150 stores various information input from the external device by the I / F unit 110 and various information calculated and generated by the control unit 120.

More specifically, various information generated by each application executed by the application execution unit 121 may be stored in the storage unit 150 and appropriately read out.

Although the configuration of the information processing apparatus 100 according to the present embodiment has been described above, the configuration of the information processing apparatus 100 is not limited to the example shown in FIG. For example, the information processing device 100 may be composed of a plurality of devices, or at least a part of the information processing device 100 may be provided in the projector 210 or the camera 310. Further, at least a part of the configuration of the information processing apparatus 100 may be provided in the server.

Further, the information processing apparatus 100 may have a function of detecting the position of a user who is writing with the digital pen 400 based on the sensing data received from the sensor 300. For example, when an RGB-IR camera in which an RGB camera and an IR camera are combined is used, the information processing apparatus 100 can acquire a visible light image and an IR image at the same time. The control unit 120 of the information processing apparatus 100 can recognize the user's position and the like from the visible light image. Further, the information processing apparatus 100 may recognize the user's position or the like based on the sensing data detected by the depth sensor such as a stereo camera or a ToF (Time Of Flight) camera.

(2-4-1. Functional configuration of writing application)
The functional configuration of the writing application, which is an example of the second application executed by the application execution unit 121, will be described with reference to FIG.

As shown in FIG. 3, the writing application 1210 mainly functions as a light emission recognition unit 1211, a coordinate conversion unit 1212, an operation detection unit 1213, and a writing screen image generation unit 1214.

-Light emission recognition unit 1211
The light emission recognition unit 1211 detects the bright spot of the IR LED as the position of the digital pen 400 (input device) based on the captured image of the projection area 211 (display area) captured by the camera 310 (imaging device) (the bright spot of the IR LED). Recognition process). The detected position coordinates of the bright spot (the light emitting position of the IR LED detected in the camera coordinate system) are stored in the storage unit 150. Further, the light emission recognition unit 1211 continuously detects the position of the detected bright spot based on the captured image continuously captured by the camera 310, that is, tracks the position of the digital pen 400.

-Coordinate conversion unit 1212
The coordinate conversion unit 1212 converts the position coordinates of the bright spot recognized by the light emission recognition unit 1211 into the display coordinate system of the projector 210. Specifically, the coordinate conversion unit 1212 converts (calculates) the position coordinates using the projection matrix. The converted position coordinates are stored in the storage unit 150. Further, the coordinate conversion unit 1212 continuously converts the position coordinates based on the position coordinates of the bright spot continuously recognized by the light emission recognition unit 1211.

-Operation detection unit 1213
The operation detection unit 1213 detects that an operation has been performed on the digital pen 400 based on the information received from the digital pen 400. For example, the operation detection unit 1213 detects that the switch 440 provided on the digital pen 400 is turned on (that is, it is in a drawing state).

-Writing screen image generation unit 1214
The writing screen image generation unit 1214 generates a writing screen image in which writing on the drawing surface by the digital pen 400 is reflected. For example, the writing screen image generation unit 1214 draws based on the bright spot (IR light position) converted by the coordinate conversion unit 1212 when the operation detection unit 1213 detects that the digital pen 400 is in the drawing state. Generate an image that maps information (handwriting image). As a result, for example, when the user touches the pen tip of the digital pen 400 on the drawing surface and draws a locus, an image to which a handwriting image showing the locus is mapped is generated. The writing screen image generation unit 1214 generates a screen image that reflects the writing on the drawing surface by the digital pen 400 and that has a transparent background.

The functional configuration of the writing application has been explained above. The writing application according to the present embodiment can identify the positions of a plurality of digital pens 400 and generate a handwriting image showing a locus drawn by each of the digital pens 400. That is, it is possible for a plurality of people to write on the drawing surface at the same time. The identification of the position of the digital pen 400 can be realized, for example, by associating the ID received from the digital pen 400 with the bright spot detected from the image captured by the camera 310. Specifically, the writing application 1210 communicates with the digital pen 400 via the I / F unit 110 of the information processing apparatus 100 to acquire an ID. Next, the writing application 1210 transmits an IR light emission command to the digital pen 400 connected by communication, and then regards a bright spot newly detected from the captured image as the position of the digital pen 400, and ID of the digital pen 400. To associate.

(2-4-2. Overlapping display of writing screen image)
In the present embodiment, even if the writing function by an input device such as a digital pen 400 is not mounted on the first application side, the writing screen image with the transparent background presented by the second application is superimposed and displayed. It is possible to make it appear as if the image (screen) presented by the first application is being written. Such superimposed display of the writing screen image will be specifically described below with reference to FIGS. 4 and 5.

FIG. 4 is a diagram showing an example of a projected image in which the writing screen image according to the present embodiment is superimposed and displayed. As shown in FIG. 4, in the projection area 211, a handwriting image 511 showing a locus drawn by a user on a drawing surface (projection area 211) in real space using a digital pen 400, and a bubble image 501 to represent a bubble. A projected image 50 including 503 is projected. The bubble images 501 to 503 are examples of depictions that clearly indicate the place (area) to be written by the user on the drawing surface, and the color, shape, size, number, and expression method are particularly limited as long as the display can distinguish the areas. Not done. For example, an announcement such as "Please write in a bubble" may be made in advance to the user. Further, when the writing is completed, an animation in which the written bubble images 501 to 503 randomly float on the screen may be displayed in the projection area 211 (details will be described later with reference to FIG. 8).

Here, the presentation of the handwriting image 511 by the digital pen 400 can be performed by the writing application which is an example of the second application. Further, the presentation of the bubble images 501 to 503 and the subsequent presentation of the animation of the bubble images 501 to 503 can be performed by the written application which is an example of the first application. The "written application" is an application on which the writing screen image presented by the writing application is superimposed and displayed. Although writing is not actually performed, in the present specification, it appears as if writing is being performed when the background presented by the writing application is located on the back side of the transparent writing screen. Called "written application". The write-to-write application may be a general application or an application having a function of cooperating with the write application.

More specifically, the projected image 50 projected on the projection area 211 has a configuration in which two images are superimposed, as shown in FIG.

FIG. 5 is a diagram illustrating superimposition of two images according to the present embodiment. The first image 500 shown in FIG. 5 is a screen of a bubble image presented by the writing application, and the second image 510 is a writing screen presented by the writing application. Since the background of the second image 510 (writing screen) is transparent, when the second image 510 (writing screen) is superimposed on the first image 500, it is as if writing on the screen of the bubble image as shown in FIG. It becomes possible to show to. In the example shown in FIG. 5, both the first image 500 and the second image 510 are displayed in full screen, but the present embodiment is not limited to this. For example, at least a part of the area of the second image may be superimposed on at least a part of the area of the first image. Further, a plurality of second images may be superimposed on the area of the first image. The plurality of second images in this case may be at least partially overlapped with each other, may be separated from each other, or may be arranged adjacent to each other.

<< 3. Operation processing >>
Subsequently, the flow of the operation processing of the information processing system according to the present embodiment will be described using a flowchart.

<3-1. First operation processing example>
FIG. 6 is a flowchart showing an example of the overall flow of the first operation process according to the present embodiment. The first operation process will be described when there is no cooperation between applications (when interprocess communication or the like described later is not performed).

As shown in FIG. 6, first, the application execution unit 121 starts the first application (written application) (step S103).

Next, the application execution unit 121 starts a second application (writing application) (step S106).

The starting order of the first application and the second application is not particularly limited, and they can be started at each timing. Specifically, it may be activated in response to an operation input by the administrator (for example, double-clicking each icon displayed on the display unit 140). Further, the second application (writing application) may be started by the application execution unit 121 in response to a specific event that occurs in the digital pen 400, such as an operation of a button provided on the digital pen 400. Unlike the first and second applications, the application execution unit 121 may be a function of a service program that is started separately, and the service program receives from the digital pen 400 via the I / F unit 110. Monitor the information. In this way, when there is no cooperation between applications, each application (first application and second application) is started based on a manual operation by a user or an administrator.

Next, the display image generation unit 122 generates a display image in which the screen presented by the second application is superimposed on the screen presented by the first application, and the display control unit 123 displays the display image from the I / F unit 110. It is output to the apparatus 200 and the display is controlled (step S109). The display image generation unit 122 and the display control unit 123 may be functions of the service program.

Next, as a function of the writing application, a writing process using the digital pen 400 is performed (step S112). As described above, the writing application generates a handwriting image showing a locus drawn on a drawing surface (projection area 211) in real space by a digital pen 400, maps it to a writing screen image, and projects it on the projection area 211. Such a writing screen image has a transparent background and is displayed superimposed on the screen presented by the application to be written. Therefore, the user substantially writes on the transparent screen (drawing surface) superimposed and displayed on the screen presented by the application to be written, and the writing is reflected on the transparent screen. It becomes possible to visually recognize the writing state on the screen presented by the application to be written.

The writing application displays an icon indicating that writing with the digital pen 400 is possible (that is, the writing application is running) and that pen input is possible on the screen presented by the writing application. It may be notified by, or it may be notified by playing voice guidance or music from a speaker (not shown).

Subsequently, when the writing end operation is performed, the writing application performs a process of saving the written image (writing screen image) in the storage unit 150 (step S115). The operation of ending the writing may be performed by the user who is writing or the administrator who is operating the information processing apparatus 100. For example, the administrator may operate a specific key on the keyboard (an example of the operation input unit 130) to end the operation, or the user operates the end icon displayed on the projected screen with the digital pen 400 to end the operation. You may let me. Alternatively, it may be automatically terminated by a timer. Further, the event may be terminated when a predetermined end event that occurs in the digital pen 400, such as the operation of a button provided on the digital pen 400, is performed. Further, it may be terminated when the service program detects that the written application presenting the screen displayed on the back of the screen presented by the writing application has terminated and the service program notifies the termination.

Further, in the present embodiment, the writing screen image is saved as a single data regardless of the image on the written side (the written image that is not actually written but appears to be overwritten on the screen). Can be done. That is, the writing screen image is managed by a second application different from the first application that handles the written image. Specifically, for example, the second image 510 shown in FIG. 5 is saved as a writing screen image. At this time, the writing application may add writing date / time information and information of the writing application to the writing screen image to be saved. Further, at this point, no change (writing) has been made in the data to the image to be written (the display screen presented by the application to be written, for example, the first image 500 shown in FIG. 5).

The application to be written may have a function of performing various processes such as compositing based on the written image saved by the writing application. In this case, the write-to-write application acquires the saved written image by an operation by the administrator or the like, and performs a process of synthesizing the written image and the written image (step S118). The written image is an image with a transparent background, and the written application can generate a written image by superimposing the written image on the written image and synthesizing the written image. Further, the control unit 120 may execute such a synthesis process by another application.

The first operation process according to the present embodiment has been described above. As a result, any application that does not implement the function of linking with the write application can be used as the write-to-write application. The user or the administrator can easily perform pen input to any application simply by starting the application for which pen input is desired (first application) and the pen input application (second application). ..

<3-2. Second operation processing example>
Next, the operation processing when there is cooperation between applications (when performing interprocess communication or the like described later) will be described.

Here, as the write-to-write application (first application), a good-luck flag display application that realizes entertainment-like writing as shown in FIG. 4 in cooperation with the write application (second application) is used. Hereinafter, a specific description will be given with reference to FIG. 7.

FIG. 7 is a sequence diagram showing an example of the operation processing of the writing display application and the writing application according to the present embodiment. As shown in FIG. 7, first, the message display application is started by a start operation by an administrator or the like (clicking the icon of the message display application arranged on the screen displayed on the display unit 140, etc.) (step S123). .. When the Good Luck Flag display application is started, the menu screen or the like presented by the Good Luck Flag display application is displayed on the display unit 140. The administrator or the like controls the operation input unit 130 to project the screen presented by the message display application on the projector 210. Alternatively, the screen presented by the message display application may be automatically projected by the projector 210 connected to the information processing apparatus 100. At this time, the screen displayed on the display unit 140 may be projected, or a part of the screen displayed on the display unit 140 may be projected.

Next, when the write function in cooperation with the write application is turned on (execution instruction) by the administrator or the like (step S129), the good-luck flag display application controls the start of the write application (execution of the process to start) (step S129). Step S132).

Next, the writing application is started (step S135). The light emission recognition unit 1211 included in the functional configuration of the writing application detects the bright spot due to the IR light emission of the digital pen 400 based on the captured image of the projection area 211 captured by the camera 310, and recognizes it as the position of the digital pen 400. (Step S138). Then, the light emission recognition unit 1211 starts tracking (a process of continuously recognizing the bright spot) of the recognized bright spot (position of the digital pen 400).

Next, the coordinate conversion unit 1212 included in the functional configuration of the writing application performs coordinate conversion for converting the bright spot position (camera coordinate system) detected from the captured image into the display coordinate system of the projector 210 (step S141).

Next, when the writing screen image generation unit 1214 included in the functional configuration of the writing application acquires information from the digital pen 400 that the switch 440 is turned on by pressing the digital pen 400 against the drawing surface, the digital pen 400 draws the image. A handwriting image showing a trajectory drawn on the surface (movement trajectory of the recognized bright spot) is mapped, and a writing screen image transparent to the background is generated (step S144).

Next, the writing application controls to display (project) the generated writing screen image from the projector 210 (step S147). More specifically, as a function of the display control unit 123, display (projection) control of the writing screen image can be performed.

On the other hand, the good-luck flag display application controls to display (project) the image to be written (step S150). The display of the image to be written may be performed in conjunction with the writing function ON shown in step S129. More specifically, as a function of the display control unit 123, display (projection) control of the image to be written is performed. Further, the image to be written is an image including a display clearly indicating the writing area, which is generated assuming writing on the screen by the digital pen 400 realized by the writing application (for example, the first image of FIG. 5). 500). In the image to be written projected on the projection area 211, the user writes in the area specified as the writing area (for example, in the area of the bubble images 501 to 503 shown in FIG. 4) with the digital pen 400. Writing with the digital pen 400 is detected by the function of the writing application and reflected in the projection area 211. Specifically, the writing screen image is superimposed and displayed on the image to be written.

Next, the writing display application turns on the writing timer when the writing time is set (step S153). The write timer may be counted after displaying the image to be written, for example. When the remaining time falls below the threshold value, such as 30 seconds remaining, the good-luck flag display application may display a countdown for counting the remaining time on the image to be written. For example, a countdown display (display of the same countdown) may be performed near each of the bubble images 501 to 503 shown in FIG. During the writing time, the display positions of the bubble images 501 to 503 do not change, and an animation may be added such that the frame portion expressing the bubbles and the shadow of the bubbles are slightly swayed.

Next, when the predetermined time elapses (step S156 / Yes), the good-luck flag display application controls to end the writing (step S159). Specifically, the good-luck flag display application notifies the writing application of the end of writing, for example, by interprocess communication. Here, as an example, writing to the image to be written is completed by a countdown, but the present embodiment is not limited to this, and for example, the administrator may end writing at any timing.

Next, when the writing application receives a notification (for example, interprocess communication) from the writing display application, the writing application saves the writing image superimposed on the written image in the storage unit 150 (step S162). The data format at the time of saving the written image is arbitrarily set. Further, the written image to be saved is assumed to be saved on one screen to which a plurality of handwritten images are mapped, for example, as in the second image 510 shown in FIG.

Next, when the end operation is performed (step S165 / Yes), the operation of the writing application ends. The end operation may be an end operation by the administrator, or may be a notification (for example, interprocess communication) from the message display application. Further, after saving the above-mentioned written image, if it is not set to continue the next writing, it may be automatically terminated. In the case of the setting for continuing the next writing, the writing application saves the written image, displays a new unwritten writing screen, and repeats the above steps S138 to S162.

On the other hand, the writing display application acquires the writing screen image saved by the writing application from the storage unit 150 (step S168) and generates a composite image (step S171). In the present embodiment, as an example of the cooperation function of the good-luck flag display application, a case where the writing application is started as shown in step S132 and a function of generating a composite image with the written image based on the writing screen image is described will be described. ..

Next, the good-luck flag display application controls to display the generated composite image (good-luck flag image) (step S174). For example, in the case of a setting for continuing the next writing, the display of the composite image is presented by an animation floating around the bubble images 501 to 503 newly displayed for the next writing, and the above steps S150 to S174 are repeated. May be. The composite image floating in the background may be displayed for a certain period of time and deleted after a certain period of time. By presenting a display in which the composite image floats around one after another each time it is written, it is possible to realize the display of the message by a large number of people. Since the writing screen image is saved separately, even after the composite image is deleted without being saved, the writing display application can read it at any time to generate and display the composite image (combined image).

Here, the generation and display of the composite image (combined image) will be specifically described with reference to FIG. FIG. 8 is a diagram illustrating a case where a composite image with a written image is generated based on a writing screen image. As shown in FIG. 8, when the writing screen image (as an example, the second image 510) saved by the writing application is acquired, the writing display application simultaneously displays the written image (as an example) displayed on the background side. The process of separating with the meaningful layout information of the first image 500) is performed. The layout information includes, for example, information on the position and contour of the bubble image that clearly indicates the range of the writing area included in the first image 500. The write-to-write application separates at least a portion of the write screen image, for example, by the contour of a bubble image showing the extent of the write area contained in the first image 500. More specifically, as shown in FIG. 8, separated images 510a to 510c separated by the contour of each bubble image are extracted from the second image 510.

Next, the message display application generates composite images 520a to 520c in which separated images are combined with each bubble image cut out from the first image 500.

Next, the message display application controls to display the composite images 520a to 520c on the projected image (first image 500b) projected by the projector 210. The message display application may present an animation in which composite images 520a to 520c are randomly floated on the screen. At this time, the message display application can appropriately adjust the size, lightness, hue, saturation, transparency, etc. of the composite images 520a to 520c. Further, in the first image 500b, bubble images 501 to 503 in which the next writing is performed are displayed. The composite images 520a to 520c may be displayed, for example, around the bubble images 501 to 503 or as a background image.

Then, when the end operation is performed by the administrator or the like (step S177 / Yes), the operation of the write-to-write application ends.

The operation processing of the writing application according to this embodiment and the writing display application linked to the writing application has been explained above. As a result, if you want to write a message by pen input, you can start the pen input application (writing application) at any time by operating the message display application. The user or the administrator does not need to load the image data or the like into the pen input application in advance, and the troublesome operation is reduced. In addition, by combining with other applications and pen input applications, the entertainment of pen input can be enhanced.

In the present embodiment, the generated composite image is expressed as a "good-luck flag", but the present disclosure is not limited to this. As an example, the write-to-write application may be an application that can enhance the entertainment of pen input in cooperation with the write application.

(Pen input status notification and end icon display)
A notification indicating that writing by the digital pen 400 is possible and an end icon for accepting a write end operation may be displayed on the screen presented by the writing application. FIG. 9 is a diagram showing an example of a notification of a pen input state and an end icon according to the present embodiment. As shown in FIG. 9, the projected image 50 may display an icon 540 indicating that writing by the digital pen 400 is possible, and an end icon 542 for ending pen input. When the end icon 542 is selected by the digital pen 400 or the like, the writing application receives the instruction to end writing and performs the writing image saving process shown in step S162. Further, the writing end display application may also acquire the input of the writing end instruction, and the processing of the above step 159 or the processing of the above step 168 and subsequent steps may be performed.

Further, the writing application may notify that writing with the digital pen 400 is possible by displaying an image 541 showing a state in which a person's silhouette, a character, CG, a person in a live-action image, or the like is writing. good. The image 541 may be a still image or a moving image. Further, the writing application may notify by text display that writing with the digital pen 400 is possible.

Further, the writing application may notify that writing by the digital pen 400 is possible by playing voice guidance or music from a speaker (not shown). Further, the writing application may notify that writing by the digital pen 400 is possible by lighting a color LED provided on the digital pen 400.

The above example can be similarly applied to other operation processing examples according to the present embodiment.

<3-2. Third operation processing example>
Next, the operation processing when the written application linked with the writing application is a general application will be described. General applications (hereinafter referred to as general applications) include, for example, text creation applications, presentation applications, spreadsheet applications, electronic document viewing / editing applications generated in a predetermined format, and photo and video viewing applications. Can be mentioned.

FIG. 10 is a flowchart showing an example of operation processing of a general application and a writing application according to the present embodiment.

As shown in FIG. 10, first, the application execution unit 121 of the information processing apparatus 100 activates a general application (written application) (step S203). The general application may be started by the operation input by the administrator. The administrator may be the same person as the user who writes using the digital pen 400.

Next, when the write function set in the general application is turned on (step S206 / Yes), the write application is started (step S209). The ON operation of the writing function is performed by, for example, from the menu screen of a general application by the administrator. Further, the writing application may be started by interprocess communication or the like from a general application.

Next, the control unit 120 controls to superimpose the screen of the writing application on the screen of the general application and display (project) it on the projection area 211 or the like (step S212). The screen of the writing application is a screen with a transparent background. Although details are omitted, as in each operation process described above, there is a screen in which the background to which the handwriting image showing the trajectory drawn on the drawing surface (projection area 211, etc.) in the real space is mapped by the digital pen 400 is transparent. , Presented by the writing application. The user can use the digital pen 400 to write on the screen of a general application projected on the projection area 211 or the like.

Next, when there is an operation to terminate the general application (step S215 / Yes), the general application determines whether or not to save the contents written by the linked writing application (step S218). Whether or not to save the written contents may be set in advance by the administrator or the user, or a pop-up display or the like may be displayed to confirm with the administrator or the user before terminating the general application.

Next, when saving the written contents (step S218 / Yes), the general application requests the writing application to save the writing screen image by interprocess communication or the like. The writing application saves the writing screen image in the storage unit 150 and ends the operation. The operation of the general application is also terminated (step S224).

Next, when the written content is not saved (step S218 / No), the general application ends the operation without making a save request to the writing application (step S221). In addition, the general application may send the operation end notification to the writing application by interprocess communication or the like.

On the other hand, when there is an operation to end the writing application (step S227 / Yes), the writing application saves the writing screen image and then ends the writing application (step S230). It should be noted that the saving of the writing screen image may be temporary saving. For example, even if the writing application is terminated first, when the general application is terminated, it is determined whether or not to save the written contents. When the written content is not saved, the general application may request the writing application to discard the temporarily saved written content by interprocess communication or the like. When saving the written contents, the destruction request is not made.

The above is the explanation of the cooperation between the general application and the writing application. As described above, even in the case of a general application in the present embodiment, when pen input is desired, a pen input application (writing application) can be started by an operation from the general application. The user or the administrator does not need to load the image data or the like into the pen input application in advance, and the troublesome operation is reduced. Further, even if the general application is set to be non-writable, writing is performed on the superimposed screen, so that the convenience is further improved. Further, since the written content is saved separately from the data on the side to be written, the written content can be read at any time to check the writing state. In addition, the written content can be easily hidden by the screen switching operation that displays the application to be written in the foreground.

Further, in the above-mentioned operation processing, it has been explained that the notification from the general application to the writing application is performed by inter-process communication or the like (inter-app communication), but the present disclosure is not limited to this. For example, the notification may be given via a service program separately started by the application execution unit 121.

<< 4. Eraser function >>
Subsequently, the eraser function according to the present embodiment will be described. The eraser function according to the present embodiment has a function of erasing only the locus drawn on the writing application side and a function of erasing the information displayed on the written application side together with the locus.

FIG. 11 is a diagram illustrating two types of eraser functions according to the present embodiment. Execution of the eraser function can be performed using, for example, a digital pen 400. When the handwriting written on the bubble image is traced with the digital pen 400 with the eraser function ON as shown in the screen example 550 of FIG. 11, only the handwriting written on the screen example 551 is erased. Alternatively, as shown in screen example 552, the handwriting and the background bubble image may be erased.

Since the background bubble image is an image presented by the writing application and the handwriting is an image presented by the writing application, for example, by reflecting the input information by the eraser function only in the writing application, only the handwriting can be erased. It is possible. When the input by the eraser function is performed by the digital pen 400, the writing application may perform an operation process of erasing the handwriting displayed at the position of the recognized digital pen 400.

On the other hand, when erasing the image presented by the background written application, the writing application may notify the writing application of the input information (eraser input coordinates, etc.) by the eraser function by interprocess communication, or the background on the writing application side. A method of painting with a color can be considered. Hereinafter, a specific description will be made with reference to FIGS. 12 and 13.

<4-1. When there is an eraser function cooperation>
FIG. 12 is a sequence diagram showing an operation process when the eraser function is linked according to the present embodiment. In FIG. 12, it is assumed that the eraser function can be added to the application to be written.

As shown in FIG. 12, when the writing application detects the eraser input that also targets the background by the digital pen 400 (step S303), the writing application notifies the writing application of the eraser input coordinates by interprocess communication or the like (step S306). Although the description is omitted in FIG. 12, the writing application side naturally performs the process of erasing the handwriting corresponding to the eraser input coordinates.

Next, the write-to-write application receives the eraser input coordinates from the write application and performs a process of erasing the image (for example, a bubble image) corresponding to the eraser input coordinates (step S309). The write-to-write application has an added function that can receive the eraser input coordinates from the write application and perform the corresponding processing (erasing processing).

This makes it possible to erase the background as well as the handwriting in real time, even for the image presented by the written application that is different from the writing application.

<4-2. When there is no cooperation with the eraser function>
On the other hand, refer to FIG. 13 for the eraser function when the application in which data cannot be edited, the application in which it is difficult to add the erasing process, and the application to which the function for linking with the writing application is not added is the written application. explain.

FIG. 13 is a flowchart showing an operation process when there is no cooperation of the eraser function according to the present embodiment. As shown in FIG. 13, the write-to-write application without the linkage function is activated according to an operation by an administrator or the like and displays an image (step S313).

On the other hand, the writing application is also started according to the operation of the administrator or the like, and the writing screen image with the transparent background is superimposed on the image of the writing application and displayed. The writing application can detect the eraser input that targets the background as well as the drawing input by the digital pen 400 (step S316).

Next, the writing application performs a process of filling the handwriting corresponding to the eraser input coordinates with the background color (step S319). The background color is the background color of the screen presented by the application to be written. For example, in the example shown in FIG. 11, white is the background color. For example, the state of screen example 552 can be realized by painting the portion traced by the digital pen 400 with white, which is the background color of the back side screen (the screen presented by the application to be written), and the handwriting and the background. You can make it look like both of them have disappeared.

Next, when the end operation by the administrator or the like is performed in the write-to-write application (step S322), the write-to-write application saves the latest data in the storage unit 150 (overwrite save, etc.) and ends the operation (step S325). ). If there is no update from the already saved data, the saving process may not be performed.

Next, the writing application acquires the saved data of the application to be written (step S328), and performs a process of synthesizing the writing screen image including the background color filling (eraser processing) with the acquired data (step S331). .. The process of synthesizing is to save the acquired data in a state in which the writing screen image including the eraser process in which the background is transparent is superimposed (may be saved in an image format). As a result, regardless of the format of the data to be written by the application to be written, it is possible to reflect the written contents including the eraser process in the data. The synthesized data is stored in the storage unit 150. Further, only the writing content (writing screen image) including the eraser processing may be stored in the storage unit 150. By superimposing it on the image of the application to be written at any time, it is possible to view the state in which the writing including the eraser processing has been performed.

If the application to be written is a general application consisting of a plurality of pages, the writing application may monitor the display screen (projected image) to detect page feed (change in the back screen). In addition, the write-to-write application saves the write screen image (may add page information) when page feed is performed, and displays a new write screen image on the display screen. good.

In addition, when the writing application side or another application is provided with a function to reflect the information of the writing screen image including the eraser processing in the data format of the writing application, the data format of the writing application is used as the compositing process. The editing process that reflects the information of the writing screen image including the eraser processing may be performed above.

<4-3. Switching the eraser function>
In the eraser function according to the present embodiment, the function of erasing only the handwriting and the function of erasing the handwriting and the background may be appropriately switched as needed.

For example, the writing application may be selected by the user using a GUI (graphic user interface). FIG. 14 is a diagram showing an example of a GUI for selecting an eraser function according to the present embodiment. In the screen example 560 of FIG. 4, an icon 561 for selecting a function for erasing the handwriting and the background and an icon 562 for selecting the function for erasing only the handwriting are displayed. Each icon makes it possible to intuitively select the eraser function. Further, each icon may be selected by using the digital pen 400 or by touching the icon by hand. The touch operation can be detected by a touch sensor, a depth sensor, or the like.

Also, the eraser function may be switched according to the switching of the drawing tool. FIG. 15 is a diagram illustrating selection of an eraser function by switching drawing tools according to the present embodiment. The screen example 570 of FIG. 15 shows an example in which the function of erasing only the handwriting is executed when the user rubs with a finger, and the screen example 572 shows a function of erasing the handwriting and the background when the user erases with the digital pen 400. Here is an example of how is executed. In this way, when the hand or finger is used, only the handwriting may be erased, and when the digital pen 400 is used, the background may be erased. As a means for detecting a hand or a finger, a finger recognition using a depth sensor or a method using a distance measuring sensor such as LiDAR can be considered. For example, it is possible to continuously detect a plane parallel to the drawing surface with LiDAR and detect the presence of a finger when the line screen is touched. Further, a touch sensor may be provided on the drawing surface.

Further, in addition to the digital pen 400, the eraser function may be switched by using a drawing tool. For example, when the digital pen 400 is used, the function of erasing only the handwriting may be executed, and when the mouse operation is performed, the function of erasing the background may be executed.

Also, the eraser function may be switched depending on the operation method. For example, when a specific place is traced once within a unit time, the function of erasing only the handwriting may be executed, and when the trace is repeated a plurality of times within a certain time, the function of erasing the background may be executed. More specifically, for example, when the same place is traced once within 1 second, the function of erasing only the handwriting may be executed, and when the trace is repeated a plurality of times, the function of erasing the background may be executed.

<< 5. Others >>
<5-1. Adjusting the position of the writing area display>
This embodiment improves usability by adjusting the display position of the image according to the profile of the user who writes when the image to be written clearly indicates the place (area) to be written by the user. be able to.

FIG. 16 is a diagram illustrating adjustment of the display position of the bubble images 581 to 583 according to the position of the user's head according to the present embodiment. In FIG. 16, bubble images 581 to 583 are displayed on the projected image 52. The bubble images 581 to 583 are examples of images presented by the application to be written and clearly indicating the place (area) to be written by the user.

For example, when the application to be written controls to divide the screen vertically and display one bubble image in each divided area in consideration of writing by a plurality of people, the position and height of the head of the user who writes. Is detected and the bubble image is controlled to be displayed at a height within reach. In the example shown in FIG. 16, the bubble images 581 to 583 are displayed at appropriate heights according to the positions and heights of the heads of the users U1 to U3. As a result, it is possible to perform appropriate display control suitable for each user even when an adult, a child, or a person in a wheelchair is in a wheelchair.

As a means for sensing the position and height of the user's head, for example, an RGB camera, a depth sensor, a line sensor for measuring height, or the like may be used. Further, the height may be calculated by a method of measuring the distance from overhead by an ultrasonic sensor.

<5-2. Change in writable area>
In this embodiment, the writable area may be changed on the screen presented by the writing application in conjunction with the UI that has meaning on the screen presented by the writing application. When the UI of the write-to-write application changes dynamically, the writable area of the write application may also change dynamically.

The writable area is an area for displaying a handwriting image written using the digital pen 400. For example, if the writable area is linked in real time to the contour of the bubble image presented by the writeable application, the write application will not be able to write outside the contour of the bubble image (the handwriting image will not be displayed). You may.

Further, even if it is possible to write in real time outside the contour of the bubble image (all the handwriting images are displayed), as described with reference to FIG. 8, the second is matched to the contour of the bubble image. When the image is separated from the image 510 and combined, the handwriting image protruding outside the outline of the bubble image may not be the target of composition.

<< 6. Supplement >>
As described above, in the information processing apparatus according to the embodiment of the present disclosure, it is possible to enhance the convenience of user input by displaying the information of user input superimposed on the screens presented by different applications.

Although the preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, the present technology is not limited to such examples. It is clear that anyone with ordinary knowledge in the art of the present disclosure may come up with various modifications or amendments within the scope of the technical ideas set forth in the claims. Is, of course, understood to belong to the technical scope of the present disclosure.

For example, the display in which the written image is superimposed on the written image is not limited to the display by projection, and the display device other than the projector 210, for example, a TV device, a display, a tablet terminal, a smartphone, a mobile phone, a PC (personal computer), or a touch panel. It may be a display, an HMD (head mount display), a wearable device, or the like.

Further, it is possible to create a computer program for exerting the functions of the information processing device 100 or the digital pen 400 on the hardware such as the CPU, ROM, and RAM built in the information processing device 100 or the digital pen 400 described above. .. Also provided is a computer-readable storage medium that stores the computer program.

Further, the effects described in the present specification are merely explanatory or exemplary and are not limited. That is, the technique according to the present disclosure may exert other effects apparent to those skilled in the art from the description of the present specification, in addition to or in place of the above effects.

The present technology can also have the following configurations.
(1)
A control unit for displaying and outputting a first screen presented by the first application and a second screen with a transparent background presented by the second application that accepts write input is provided.
The control unit
Control is performed to superimpose and display the second screen on the first screen.
Image processing using the second screen including the display of the write input and the first screen is performed.
Information processing equipment.
(2)
The information processing device according to (1) above, wherein the control unit performs a process of generating a composite image in which the second screen is combined with the first screen.
(3)
The control unit sets an image obtained by separating at least a part from the second screen according to the layout information extracted from the first screen, as a part of the first screen corresponding to the layout information. The information processing apparatus according to (1) or (2) above, which performs a process of generating a composite image combined with an image.
(4)
The information processing apparatus according to (3), wherein the layout information includes information on the outline of an image that clearly indicates a writing area included in the first screen.
(5)
The second application recognizes the position of the input device, accepts the locus drawn on the drawing surface by the input device as the write input, and displays the handwriting image showing the locus on the second screen. The information processing apparatus according to any one of (1) to (4).
(6)
The information processing device according to (5) above, wherein the control unit controls to display a display screen on which the second screen is superimposed on the first screen on the drawing surface.
(7)
The information processing device according to (6), wherein the control unit controls to project the display screen onto the drawing surface.
(8)
The information processing apparatus according to any one of (1) to (7) above, wherein the first application controls the activation of the second application.
(9)
The information processing apparatus according to (8), wherein the first application controls the storage of the second screen presented by the second application.
(10)
The control unit performs image processing using the second screen and the first screen after the writing input is completed and the second screen is saved. The information processing apparatus according to any one of ().
(11)
The information processing apparatus according to any one of (1) to (10) above, wherein the second application accepts an input for erasing the display of the write input on the second screen.
(12)
The second application performs a process of filling a target portion with a background color of the first screen when receiving a display of the write input and an input for erasing the display of the corresponding first screen. The information processing apparatus according to (11).
(13)
When the second application receives the display of the write input and the input for erasing the display of the corresponding first screen, the coordinate information indicating the target portion of the input for erasing the display to the first application. The information processing apparatus according to (11) above.
(14)
The processor,
Displaying and outputting the first screen presented by the first application and the second screen with a transparent background presented by the second application that accepts write input.
Controlling the display of the second screen superimposed on the first screen is performed.
Performing image processing using the second screen including the display of the write input and the first screen, and
Information processing methods, including.
(15)
Computer,
The first screen presented by the first application and the second screen with a transparent background presented by the second application that accepts write input are made to function as a control unit for displaying and outputting.
The control unit
Control is performed to superimpose and display the second screen on the first screen.
A program that performs image processing using the second screen including the display of the write input and the first screen.

100 Information processing device 110 I / F unit 120 Control unit 121 Application execution unit 122 Display image generation unit 123 Display control unit 130 Operation input unit 140 Display unit 150 Storage unit 200 Display device 210 Projector 300 Sensor 310 Camera 400 Digital pen 410 Communication module 420 Control unit 430 IR LED
440 switch

Claims (15)

1. A control unit for displaying and outputting a first screen presented by the first application and a second screen with a transparent background presented by the second application that accepts write input is provided.
   The control unit
   Control is performed to superimpose and display the second screen on the first screen.
   Image processing using the second screen including the display of the write input and the first screen is performed.
   Information processing equipment.
2. The information processing device according to claim 1, wherein the control unit performs a process of generating a composite image in which the second screen is combined with the first screen.
3. The control unit sets an image obtained by separating at least a part from the second screen according to the layout information extracted from the first screen, as a part of the first screen corresponding to the layout information. The information processing apparatus according to claim 1, wherein a process of generating a composite image combined with an image is performed.
4. The information processing apparatus according to claim 3, wherein the layout information includes information on the outline of an image that clearly indicates a writing area included in the first screen.
5. The second application recognizes the position of the input device, accepts the locus drawn on the drawing surface by the input device as the write input, and displays the handwriting image showing the locus on the second screen. Item 1. The information processing apparatus according to Item 1.
6. The information processing device according to claim 5, wherein the control unit controls to display a display screen on which the second screen is superimposed on the first screen on the drawing surface.
7. The information processing device according to claim 6, wherein the control unit controls to project the display screen onto the drawing surface.
8. The information processing device according to claim 1, wherein the first application controls the activation of the second application.
9. The information processing device according to claim 8, wherein the first application controls the storage of the second screen presented by the second application.
10. The information according to claim 1, wherein the control unit performs image processing using the second screen and the first screen after the writing input is completed and the second screen is saved. Processing equipment.
11. The information processing device according to claim 1, wherein the second application accepts an input for erasing the display of the write input on the second screen.
12. When the second application receives the display of the write input and the input for erasing the display of the corresponding first screen, the second application performs a process of filling the target portion with the background color of the first screen. Item 11. The information processing apparatus according to Item 11.
13. When the second application receives the display of the write input and the input for erasing the display of the corresponding first screen, the coordinate information indicating the target portion of the input for erasing the display to the first application. The information processing apparatus according to claim 11.
14. The processor,
    Displaying and outputting the first screen presented by the first application and the second screen with a transparent background presented by the second application that accepts write input.
    Controlling the display of the second screen superimposed on the first screen is performed.
    Performing image processing using the second screen including the display of the write input and the first screen, and
    Information processing methods, including.
15. Computer,
    The first screen presented by the first application and the second screen with a transparent background presented by the second application that accepts write input are made to function as a control unit for displaying and outputting.
    The control unit
    Control is performed to superimpose and display the second screen on the first screen.
    A program that performs image processing using the second screen including the display of the write input and the first screen.

Images