WO2018235128A1

WO2018235128A1 - Display editing apparatus, display editing method, and display editing program

Info

Publication number: WO2018235128A1
Application number: PCT/JP2017/022519
Authority: WO
Inventors: 喬之築谷
Original assignee: 三菱電機株式会社
Priority date: 2017-06-19
Filing date: 2017-06-19
Publication date: 2018-12-27
Also published as: DE112017007535B4; JPWO2018235128A1; JP6671549B2; DE112017007535T5

Abstract

This display editing apparatus is provided with: a captured image acquisition unit (111) that acquires an image obtained by imaging an object by a camera (101); a figure acceptance unit (112) that accepts first figure information about a first figure drawn on the image; a positional relationship acquisition unit (113) that acquires the positional relationship between the object and the camera when the image is captured; a coordinate conversion unit (115) that calculates second figure information on the basis of the positional relationship acquired by the positional relationship acquisition unit (113) and the first figure information; and an instruction unit (116) that outputs the second figure information calculated by the coordinate conversion unit (115) to a drawing device that draws on the object.

Description

Display editing apparatus, display editing method, and display editing program

The present invention relates to a display editing apparatus that calculates graphic information for a drawing apparatus that performs drawing on an object.

Conventionally, there is known a technique of drawing a figure on the surface of an object (hereinafter, also referred to as an “object”) according to some control. The figure drawn on the object in this manner is the position at which the person who visually recognizes the figure (hereinafter referred to as the "viewer") faces the surface of the object on which the figure is drawn (hereinafter If you look at the figure from a position (hereinafter referred to as “non-facing position”) where the viewer deviates from the facing position (not just when viewing the figure from It may be drawn so as to be visible (hereinafter simply referred to as "desired figure").
For example, in Patent Document 1, an acquisition unit for acquiring an image to be displayed, a display unit for displaying the image to be displayed obtained by the acquisition unit on a display surface, and an OSD for the image to be displayed (On Screen Display) When an instruction to superimpose an image is input, an OSD image having a different shape is superimposed on the image to be displayed according to the position of the operator who has input the instruction, and is displayed on the display surface. And a control means for controlling the display means.

JP, 2013-83755, A

According to the technology disclosed in Patent Document 1, an operator who is at a non-facing position with respect to the display surface can visually recognize, for example, a rectangular OSD image as a desired figure.
However, when editing the graphic information for instructing the drawing apparatus to draw so that the viewer can visually recognize the desired figure when viewing the figure drawn on the object from the non-facing position, the figure It is necessary for a person editing the information to edit while imagining a figure when the figure is visually recognized from the non-facing position, and there is a problem that such editing work is difficult.

The present invention has been made to solve the above-described problems, and a drawing apparatus that allows a viewer to visually recognize a desired figure when looking at a figure drawn on an object from a non-facing position. It is an object of the present invention to provide a display editing apparatus, a display editing method, and a display editing program capable of editing graphic information for instructing drawing.

A display editing apparatus according to the present invention includes: a captured image acquisition unit that acquires an image obtained by capturing an object by a camera; a graphic reception unit that receives first graphic information related to a first graphic drawn on the image; The second graphic information is calculated from the positional relationship acquisition unit that acquires the positional relationship between the object and the camera when the image is captured, the positional relationship acquired by the positional relationship acquisition unit, and the first graphic information. A coordinate conversion unit, and an instruction unit for outputting the second graphic information calculated by the coordinate conversion unit to a drawing device for drawing on an object.

According to the present invention, the graphic information for instructing the drawing apparatus to draw is edited so that a desired figure can be visually recognized when the viewer looks at the figure drawn on the object from the non-facing position. Can.

FIG. 1 is a diagram showing an exemplary configuration of a display editing system according to a first embodiment. FIG. 1 is a block diagram showing a configuration example of a display editing device of a first embodiment. FIG. 1 is a block diagram showing a configuration example of a projector of a first embodiment. 5 is a flowchart for explaining the operation of the display editing apparatus of the embodiment 1; In Embodiment 1, it is a figure explaining an example of the use scene of a display editing apparatus. 6A and 6B are diagrams showing an example of the hardware configuration of the display / editing apparatus according to Embodiment 1 of the present invention. FIG. 13 is a block diagram showing an example of configuration of a display / editing apparatus of a second embodiment. FIG. 13 is a flow chart for explaining the operation of the display editing apparatus of the embodiment 2; FIG. In Embodiment 2, it is a figure which shows an example of the screen of the display part on which the guide and the 1st figure after complementation which the figure complementation part was displayed were displayed. FIG. 18 is a block diagram showing an example of configuration of a display / editing apparatus of a third embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
Embodiment 1
The display editing apparatus is a graphic for instructing the drawing apparatus to draw so that a desired figure can be visually recognized when the viewer looks at the figure drawn on the object by the drawing apparatus from the non-facing position. Edit the information The term "graphic" in this specification includes not only simple geometrical patterns but also characters, symbols and the like. The graphic may be a still image or a moving image.
In the following description, as an example, the display editing apparatus is mounted on a tablet PC (Personal Computer) including a camera and a touch panel. Further, the drawing device is assumed to be a projector attached to a car. The projector is a projector that draws a figure on an object by projecting light. Further, the figure drawn on the object is a still image, and the object is a road surface which is a projection surface on which light is projected by the projector. Further, it is assumed that the assumed viewer is a person on the road around the road surface.
The user using the display editing apparatus according to the first embodiment is a still image of a desired shape (hereinafter referred to as “visible image”) that can be viewed when a viewer on the road around the road looks at the still picture projected on the road by the projector. What is simply referred to as “desired still image”) is drawn using a tablet PC, and the display editing apparatus edits the graphic information for instructing the projector to project based on the still image drawn by the user I assume. When the user draws a still image using a tablet PC, a captured image obtained by capturing an image of the road surface from the viewpoint of the viewer in advance is displayed on the tablet PC by the camera, and the user sees the viewer's viewpoint A desired still image as viewed from the point of view can be drawn at a desired position on the captured image displayed on the tablet PC, with the shape as it looks.

FIG. 1 is a diagram showing a configuration example of a display editing system according to the first embodiment.
The tablet PC 100 causes the display unit 102 (described later) to display an image captured by a camera 101 (described later) provided on the back surface. The display unit 102 is a touch panel display.
The user draws a desired still image by touching the display unit 102 with a finger or a so-called stylus. The display editing apparatus 1 edits the graphic information based on the still image input by the user drawing on the display unit 102 and outputs the graphic information to the projector 2, and the projector 2 applies the still image indicated by the graphic information to the road surface. Project.
The tablet PC 100 and the projector 2 may be able to communicate with each other by any means, whether wired or wireless.

FIG. 2 is a block diagram showing an example of the configuration of the display editing apparatus 1 according to the first embodiment.
As shown in FIG. 2, the display editing apparatus 1 includes a captured image acquisition unit 111, a figure reception unit 112, a positional relationship acquisition unit 113, a figure position determination unit 114, a coordinate conversion unit 115, and an instruction unit 116. , A display control unit 117, an audio output control unit 118, and a storage unit 119.

The captured image acquisition unit 111 acquires a captured image in which the camera 101 captures a road surface.
The captured image acquisition 11 outputs the acquired captured image to the positional relationship acquisition unit 113 and the display control unit 117. Hereinafter, the captured image acquired by the captured image acquisition unit 111 and output to the positional relationship acquisition unit 113 and the display control unit 117 is simply referred to as an image.

The graphic receiving unit 112 receives graphic information (hereinafter referred to as “first graphic information”) related to a graphic drawn on the image displayed on the display unit 102 (hereinafter referred to as “first graphic”).
When the image is displayed on the display unit 102, the user operates, for example, the touch panel to draw a first figure on the image as a desired still image. The graphic receiving unit 112 receives first graphic information on a first graphic drawn by the user.
The figure reception unit 112 outputs the received first figure information to the figure position determination unit 114.

The positional relationship acquisition unit 113 acquires the positional relationship between the road surface and the camera 101 when the captured image acquired by the captured image acquisition unit 111 is captured. Further, the positional relationship acquisition unit 113 acquires the positional relationship between the projector 2 and the camera 101 when the captured image acquired by the captured image acquisition unit 111 is captured.
The positional relationship acquisition unit 113 outputs the acquired positional relationship information to the figure position determination unit 114.

The figure position determination unit 114 outputs the information on the positional relationship between the road surface and the camera 101, the information on the positional relationship between the projector 2 and the camera 101, which is output from the positional relationship acquisition unit 113, and Based on the graphic information of 1, it is determined to which position on the road surface the first graphic is associated.
At this time, when the first figure is outside the road surface, the figure position determination unit 114 outputs the error information indicating that the first figure can not be projected onto the road surface, the display control unit 117 or the voice output. It is output to the control unit 118.
When the first figure is on the road surface, the figure position determination unit 114 combines the first figure information with the information on the positional relationship between the road surface and the camera 101, and the information on the positional relationship between the projector 2 and the camera 101. It is output to the coordinate conversion unit 115.

When the first graphic information and the like are output from the graphic position determination unit 114, the coordinate conversion unit 115 outputs information on the positional relationship between the road surface and the camera 101, information on the positional relationship between the projector 2 and the camera 101, The coordinate of the first graphic information is converted from the graphic information of 1, and the graphic information (hereinafter, “second”) regarding the graphic to be projected onto the road surface by the projector 2 (hereinafter referred to as “second graphic”) Calculate “graphic information”. The second figure is a figure that is viewed by the viewer without changing the shape of the first figure as viewed from the camera position when the captured image acquired by the captured image acquisition unit 111 is captured. It is a figure to be projected by the projector 2 in order to draw on an object.
The coordinate conversion unit 115 outputs the calculated second graphic information to the instruction unit 116.

The instruction unit 116 outputs the second graphic information calculated by the coordinate conversion unit 115 to the projector 2 that projects the graphic on the road surface, and the projector 2 outputs the second graphic indicated by the second graphic information. , Project on the road surface.

The display control unit 117 causes the display unit 102 to display the image acquired by the captured image acquisition unit 111. In addition, when error information indicating that the first figure can not be projected onto the road surface is output from the graphic position determination unit 114, the display control unit 117 causes the display unit 102 to respond to the error information such as an error message. Display information.

When the voice output control unit 118 outputs error information indicating that the first figure can not be projected onto the road surface from the figure position determination unit 114, the voice output device 103 generates a buzzer sound, an error message, or the like. , Output sound or sound according to the error information. In the first embodiment, the audio output device 103 is a speaker provided in the tablet PC 100.

The storage unit 119 stores information related to the projector, such as the position or posture of the projector 2. The storage unit 119 also stores information on the shape of the road surface, such as unevenness.
The information to be stored in the storage unit 119 may be set in advance by the user or the like, or with regard to the position of the projector 2 or the like, when the display editing device 1 is used as described later. It may be set based on the information detected by itself.
In the first embodiment, the storage unit 119 is included in the display editing apparatus 1. However, the present invention is not limited to this. The storage unit 119 can refer to the display editing apparatus 1 outside the display editing apparatus 1. It may be prepared in place.

FIG. 3 is a block diagram showing a configuration example of the projector 2 according to the first embodiment.
As shown in FIG. 3, the projector 2 includes a drawing instruction receiving unit 21 and a drawing unit 22.
The drawing instruction receiving unit 21 receives the second graphic information output from the display editing apparatus 1.
The drawing instruction receiving unit 21 outputs the received second graphic information to the drawing unit 22.
The drawing unit 22 projects the second graphic indicated by the second graphic information received by the drawing instruction receiving unit 21 on the road surface.

Next, the operation of the display editing apparatus 1 according to the first embodiment will be described.
FIG. 4 is a flowchart for explaining the operation of the display editing device 1 according to the first embodiment.
Here, FIG. 5 is a diagram for explaining an example of a use scene of the display editing apparatus 1 in the first embodiment.
As shown in FIG. 5, the user picks up the tablet PC 100 and observes the projector 2, the automobile 51 to which the projector 2 is attached, and the periphery thereof through the tablet PC 100. Furthermore, the user uses the camera of the tablet PC 100 to, from the viewpoint position where the viewer views the still image projected by the projector 2, the road surface on which the still image is to be drawn, the projector 2, and To capture an image at least including:
The projector 2 is in a state capable of emitting an arbitrary still image to the road surface.
Assuming that the state is as described above, the display editing device 1 performs the operation as described in the flowchart of FIG. 4.

The figure reception unit 112 waits until the first figure information is received (in the case of “NO” in step ST401), and when the first figure information is received, the received first figure information is sent to the figure position determination unit 114. Output.
When the user operates the touch panel to start drawing the first graphic, the graphic receiving unit 112 receives the drawn first graphic information at any time, and thereafter, the user draws the first graphic even by a small amount. Then, each time the first graphic is drawn, the graphic receiving unit 112 continues to receive the first graphic information.

The positional relationship acquisition unit 113 acquires the positional relationship between the road surface and the camera 101. Further, the positional relationship acquisition unit 113 acquires the positional relationship between the projector 2 and the camera 101 (step ST402).
Specifically, for example, a marker is installed in advance at a predetermined position in the space within the imaging range of the camera 101, and the positional relationship acquisition unit 113 detects the marker in the video to detect the road surface. And the camera 101, and the three-dimensional positional relationship between the projector 2 and the camera 101 is acquired. Note that this is merely an example, and the positional relationship acquisition unit 113 may acquire the positional relationship between the road surface and the camera 101, and between the projector 2 and the camera 101, using another existing method. For example, the positional relationship acquisition unit 113 may obtain the positional relationship using feature quantities in the space, or may obtain the positional relationship using an existing application or the like that recognizes the surrounding three-dimensional space. . Alternatively, the three-dimensional tracker may be attached to the tablet PC 100, and the positional relationship acquisition unit 113 may acquire the positional relationship from the measurement information of the three-dimensional tracker. The display editing apparatus 1 stores in advance information on the mounting position of the projector 2 on the vehicle and the mounting attitude, and information on the shape of the road surface when the road surface includes unevenness and the like.
The positional relationship acquisition unit 113 outputs the acquired positional relationship information to the figure position determination unit 114.

The graphic position determination unit 114 is output from the graphic reception unit 112, the information of the positional relationship between the road surface and the camera 101, the information of the positional relationship between the projector 2 and the camera 101, which is output from the positional relationship acquisition unit 113. Based on the first graphic information, it is determined which position on the road surface the first graphic is associated with (step ST403).

Then, the figure position determination unit 114 determines whether the first figure is on the road surface (step ST404), and if it is outside the road surface (in the case of “NO” in step ST404), the first Error information indicating that the figure can not be projected onto the road surface is output to the display control unit 117 or the voice output control unit 118 (step ST407).
When the graphic position determination unit 114 outputs the error information, the display control unit 117 causes the display unit 102 to display information corresponding to the error information, such as an error message. Further, when the error information is output from the figure position determination unit 114, the voice output control unit 118 outputs a sound or voice according to the error information, such as a buzzer sound or an error message, to the voice output device 103. Let The figure position determination unit 114 may output the error information to both the display control unit 117 and the audio output control unit 118.

On the other hand, when the first figure is on the road surface (in the case of “YES” in step ST404), the figure position determination unit 114 sets the first figure information to the information on the positional relationship between the road surface and the camera 101; The coordinate conversion unit 115 outputs information on the positional relationship between the projector 2 and the camera 101.

The coordinate conversion unit 115 uses the information on the positional relationship between the road surface and the camera 101, the information on the positional relationship between the projector 2 and the camera 101, and the first graphical information output by the graphic position determination unit 114 in step ST404. The projector 2 calculates second graphic information on a second graphic to be projected on the road surface (step ST405).
The coordinate conversion unit 115 outputs the calculated second graphic information to the instruction unit 116.

The instructing unit 116 outputs the second graphic information calculated by the coordinate conversion unit 115 in step ST405 to the projector 2 (step ST406).
In the projector 2, the drawing instruction receiving unit 21 receives the second graphic information output from the instructing unit 116, and the drawing unit 22 projects the second graphic indicated by the second graphic information on the road surface.

The figure reception unit 112 subsequently determines whether or not the first figure information is received (step ST408), and when the first figure information is received (in the case of “YES” in step ST408), the process returns to step ST402. Repeat the subsequent processing.
If the first graphic information is not received ("NO" in step ST408), the process ends. The case where the first graphic information is not received means, for example, when new first graphic information is not inputted for a predetermined time, or when the user performs an operation of ending the input process of the first graphic information. It is.

Generally, for example, in a situation where a projection mapping or the like is created, the user has a procedure of editing a figure or the like to be displayed on a personal computer and projecting the figure or the like on a projector. On a personal computer, in many cases, a user edits a figure to be displayed using an image editing software etc. in a state where it is viewed from the directly facing position, but when projecting a created figure, what kind of It can only be imagined by the user if it looks like, for example, editing is performed while minding the appearance when viewed from the non-facing position, making the task of creating a figure difficult for the user. Furthermore, since the scale is naturally different on the screen of the personal computer and on the real space, when the figure displayed on the screen of the personal computer is actually displayed, it looks different from what the user expected. There is also a fear.

On the other hand, in the first embodiment, as described above, when the viewer looks at the figure drawn on the object from the non-facing position, the user desires the desired figure to be made visible by the viewer, It is possible to draw on the tablet PC 100 as the first figure as it is. As a result, it becomes easy for the user to edit the figure, and the quality of the edited figure (second figure) is also improved. Furthermore, since the input from the tablet PC 100 can be performed on an image obtained by imaging an object on which a graphic is to be actually projected, the user can also work while confirming the actual sense of scale. The image of the figure desired to be projected does not shift, and high-quality figure can be efficiently projected.

6A and 6B are diagrams showing an example of the hardware configuration of the display editing device 1 according to the first embodiment of the present invention.
In the first embodiment of the present invention, the captured image acquisition unit 111, the figure reception unit 112, the positional relationship acquisition unit 113, the figure position determination unit 114, the coordinate conversion unit 115, the display control unit 117, and audio output. Each function of the control unit 118 is realized by the processing circuit 601. That is, the display editing apparatus 1 converts the first graphic into the second graphic based on the acquired captured image and the first graphic information received from the user, and performs processing for performing control to project the first graphic. Prepare.
The processing circuit 601 may be dedicated hardware as shown in FIG. 6A or may be a CPU (Central Processing Unit) 605 that executes a program stored in the memory 606 as shown in FIG. 6B.

When the processing circuit 601 is dedicated hardware, the processing circuit 601 may be, for example, a single circuit, a complex circuit, a programmed processor, a parallel programmed processor, an application specific integrated circuit (ASIC), an FPGA (field-programmable) Gate Array) or a combination thereof is applicable.

When the processing circuit 601 is the CPU 605, the captured image acquisition unit 111, the figure reception unit 112, the positional relationship acquisition unit 113, the figure position determination unit 114, the coordinate conversion unit 115, the instruction unit 116, and the display control unit 117. Each function of the audio output control unit 118 is realized by software, firmware, or a combination of software and firmware. That is, the captured image acquisition unit 111, the graphic reception unit 112, the positional relationship acquisition unit 113, the graphic position determination unit 114, the coordinate conversion unit 115, the display control unit 117, and the audio output control unit 118 A hard disk drive (602), a CPU 605 that executes a program stored in a memory 606 or the like, and a processing circuit such as a system LSI (Large-Scale Integration). The programs stored in the HDD 602, the memory 606, and the like are the captured image acquisition unit 111, the figure reception unit 112, the positional relationship acquisition unit 113, the figure position determination unit 114, the coordinate conversion unit 115, and the instruction unit 116. It can also be said that the computer causes the computer to execute the procedures and methods of the display control unit 117 and the audio output control unit 118. Here, the memory 606 is, for example, a non-volatile memory such as a random access memory (RAM), a read only memory (ROM), a flash memory, an erasable programmable read only memory (EPROM), and an electrically erasable programmable read only memory (EEPROM). Semiconductor memory, magnetic disk, flexible disk, optical disk, compact disk, mini disk, DVD (Digital Versatile Disc), and the like.

Note that the captured image acquisition unit 111, the graphic reception unit 112, the positional relationship acquisition unit 113, the graphic position determination unit 114, the coordinate conversion unit 115, the instruction unit 116, the display control unit 117, and the audio output control unit For each function of 118, a part may be realized by dedicated hardware and a part may be realized by software or firmware. For example, the processing circuit 601 as dedicated hardware realizes the function of the captured image acquisition unit 111, and the graphic reception unit 112, the positional relationship acquisition unit 113, the graphic position determination unit 114, and the coordinate conversion unit 115 The functions of the instruction unit 116, the display control unit 117, and the audio output control unit 118 can be realized by the processing circuit reading and executing a program stored in the memory 606.
The storage unit 119 uses, for example, the HDD 602. Note that this is merely an example, and the storage unit 119 may be configured by a DVD, a memory 606, and the like.
The display editing apparatus 1 further includes an input interface device 603 and an output interface device 604 that communicate with an external device such as the camera 101, the display unit 102, or the projector 2. For example, the captured image acquisition unit 111 acquires a captured image captured by the camera 101 using the input interface device 603. Also, for example, the graphic receiving unit 112 acquires first graphic information by an input operation of the user using the input interface device 603. Also, for example, the instruction unit 116 transmits the second graphic information to the projector 2 using the output interface device 604.

As described above, according to the first embodiment, the display editing apparatus 1 relates to a captured image acquisition unit 111 that acquires an image obtained by capturing an object by the camera 101, and a first figure drawn on the image. The positional relationship acquisition unit 113 that acquires the positional relationship between the object when the image is captured and the camera, the positional relationship acquired by the positional relationship acquisition unit 113, and the graphic reception unit 112 that receives the first graphic information. A coordinate conversion unit 115 that calculates second graphic information from the first graphic information, and an instruction to output the second graphic information calculated by the coordinate conversion unit 115 to a drawing device that draws on the object And the unit 116. Therefore, it is possible to edit the graphic information for instructing the drawing apparatus to draw so that a desired figure can be recognized when the viewer looks at the figure drawn on the object from the non-facing position.
In the first embodiment, the position of the camera when photographing an object can be any position. Therefore, in the first embodiment, in particular, the drawing apparatus is instructed to draw so that a desired figure is recognized when the viewer looks at the figure drawn on the object from the non-facing position. Although effective, a figure for instructing the drawing device to draw a figure drawn on the object, including a case where the viewer wishes to recognize a desired figure when viewed from the “right-facing position” Information can be edited.

Second Embodiment
In the first embodiment, the user often draws the second figure projected from the projector 2 in order to draw the first figure in a state where the surface of the object on which the figure is projected is viewed from the non-facing position. The figure does not become symmetrical when viewed from the projector 2 side or the car side. However, depending on the purpose of the figure projected by the projector 2 on the road surface, it may be desirable to make the figure symmetrical when viewed from the projector 2 or the car side. For example, a pedestrian or the like around a car is assumed as a viewer by a figure projected from the projector 2 attached to the car, and in the case of an application that causes the pedestrian or the like to visually recognize the traveling direction of the car It is desirable to project from the projector 2 a figure that is symmetrical when viewed from the projector 2 side or the car side in order to enable the person etc. to accurately view the traveling direction of the car.
In the second embodiment, even when the user draws the first graphic while looking at the surface of the object on which the graphic is projected from the non-facing position, the user sees from the position of the assumed viewer. An embodiment will be described in which a desired figure which is easy to visually recognize and which is symmetrical when viewed from the projector 2 side or the car side can be projected from the projector as a second figure.
Also in the second embodiment, as in the first embodiment, as an example, the display editing apparatus 1a (described later) is mounted on the tablet PC 100, and the user is attached to a car by a viewer on the road around the road surface. A desired still image visible when viewing a still image projected on the road surface by the projector 2 is drawn using the tablet PC 100, and the display editing apparatus based on the still image drawn by the user It is assumed that graphic information for instructing the projector 2 to project is edited.

The configuration of the display editing system according to the second embodiment is the same as the configuration described with reference to FIG. 1 in the first embodiment, and therefore redundant description will be omitted. Further, the configuration of the projector 2 according to the second embodiment is the same as the configuration described with reference to FIG. 3 in the first embodiment, and thus the redundant description will be omitted.

FIG. 7 is a block diagram showing a configuration example of the display editing device 1a according to the second embodiment.
In FIG. 7, the same components as those described with reference to FIG. 2 in the first embodiment are denoted by the same reference numerals, and the description thereof will not be repeated.
The display editing apparatus 1a differs from the display editing apparatus 1 according to the first embodiment in that the display editing apparatus 1a further includes a figure complementing unit 120.

The graphic complementing unit 120 causes the display control unit 117 to display a guide superimposed on the image displayed on the display unit 102 at an arbitrary timing before the drawing of the first graphic information by the user starts. Further, when the first graphic information received by the graphic receiving unit 112 is acquired, the first graphic is complemented so as to be symmetrical with respect to the projector 2 side or the automobile side based on the first graphic information, and the complementation is performed. The graphic information of the later first graphic (hereinafter referred to as “first graphic information after complementation”) is created.
For example, the figure complementing unit 120 displays, as a guide, an image of a grid that is symmetrical with respect to the projector 2 side or the vehicle side. In the second embodiment, the figure complementing unit 120 superimposes and displays the guide on the position of the road surface in the image.
The figure complementing unit 120 outputs the first figure information after complementation to the figure position determination unit 114. Further, the figure complementing unit 120 also outputs the first figure information after complementation to the display control unit 117, superimposes it on the image displayed by the display unit 102, and causes the first figure information after complementation to be displayed. (In FIG. 7, the description of connection lines for outputting information from the figure complementing unit 120 to the display control unit 117 is omitted.)
The figure position determination unit 114 outputs the information on the positional relationship between the road surface and the camera 101, the information on the positional relationship between the projector 2 and the camera 101, and the complementation output from the figure complementing unit 120, which are output from the positional relationship acquisition unit 113. Based on the post-first graphic information, it is determined which position on the road surface the first graphic after complementation is associated with.

The hardware configuration of the display editing apparatus 1a is the same as the configuration described with reference to FIGS. 6A and 6B in the first embodiment, and thus the redundant description will be omitted.
The figure complementing unit 120 includes a captured image acquisition unit 111, a figure reception unit 112, a positional relationship acquisition unit 113, a figure position determination unit 114, a coordinate conversion unit 115, an instruction unit 116, a display control unit 117, and an audio output control unit 118. Similarly, it is realized by the processing circuit 601.

Next, the operation of the display editing apparatus 1a according to the second embodiment will be described.
FIG. 8 is a flowchart for explaining the operation of the display editing device 1a according to the second embodiment.
The specific operations of steps ST 801 and 803 to 809 in FIG. 8 are the same as the specific operations of steps ST 401 to 408 in FIG. 4 described in the first embodiment, and therefore the description thereof will not be repeated.
In the second embodiment, only the operation of step ST802 of FIG. 8 is added to the operation described with reference to FIG. Here, it is assumed that the guide is already displayed on the display unit 102 by the figure complementing unit 120 before the figure receiving unit 112 receives the first figure information in ST801.

In step ST801, when the graphic receiving unit 112 receives the first graphic information, the graphic receiving unit 112 outputs the received first graphic information to the graphic complementing unit 120.
The figure complementing unit 120 complements the first figure so as to be symmetrical on the basis of the first figure information output from the figure accepting unit 112, and creates the first figure information after complementation.
The figure complementing unit 120 outputs the first figure information after complementation to the figure position determination unit 114. Further, the figure complementing unit 120 also outputs the complemented first figure information to the display control unit 117. When acquiring the first figure information after complementation, the display control unit 117 causes the first figure after complementation indicated by the first figure information after complementation to be superimposed on the image displayed by the display unit 102.

Here, FIG. 9 is a diagram showing an example of a screen of the display unit 102 on which the guide and the first figure after complementation are displayed in the second embodiment.
In FIG. 9, as an example, it is assumed that the user inputs an oblique line (9a in FIG. 9) as the first figure.
The figure complementing unit 120 complements the oblique line (9b in FIG. 9) so that the first figure input by the user is symmetrical with respect to the projector 2 side or the car side.

The positional relationship acquisition unit 113 acquires the positional relationship between the road surface and the camera 101, and the projector 2 and the camera 101 (step ST803), and the graphic position determination unit 114 outputs the road surface and the camera output from the positional relationship acquisition unit 113. The first figure after complementation is based on the information on the positional relationship with 101, the information on the positional relationship between the projector 2 and the camera 101, and the first complemented first graphic information output from the graphic complement unit 120. It is determined to which position on the road surface it corresponds (step ST804).

In the second embodiment described above, the figure complementing unit 120 displays the grid as a guide, but this is merely an example. The figure complementing unit 120 may display, as a guide, arbitrary information for assisting an input to a user who intends to edit a figure that is symmetrical when viewed from the projector 2 side or the car side.
Also, the figure complementing unit 120 does not display the guide, but complements the first figure so that the first figure is symmetrical left and right so that only the complemented first symmetrical figure is displayed. It is also good.

In the second embodiment, the figure complementing unit 120 complements the first figure so that the first figure is symmetrical. However, the method of complementation is not limited to the symmetrical. The figure complementing unit 120 may complement the first figure so as to be, for example, vertically symmetrical or vertically symmetrical as viewed from the projector side or the automobile side, or may be complemented as being vertically symmetrical It may be complemented so that it becomes, or complemented so that it becomes point-symmetrical.

As described above, according to the second embodiment, the first figure is complemented so that the first figure is symmetrical based on the first figure information received by the figure accepting unit 112, and the first after interpolation is performed. The coordinate conversion unit 115 generates the second figure from the positional relation acquired by the positional relation acquisition unit 113 and the first figure information after complementation generated by the figure complement unit 120. It was made to calculate figure information. Thus, even when the user creates a figure from an arbitrary viewpoint, the figure which is symmetrical as viewed from the drawing device can be displayed, and a figure of high quality can be created. .

Third Embodiment
In the first and second embodiments, for example, it is assumed that the car attached with the projector 2 is at a standstill, the position of the projector 2 is also moved as the car travels, or the travel state of the car changes The situation related to the projector 2 was not supposed to change.
In the third embodiment, even when the situation related to the projector 2 changes, an embodiment in which the projector 2 can project a figure desired by the user according to the change of the situation will be described.
Also in the third embodiment, as in the first and second embodiments, as an example, the display editing apparatus 1b (described later) is mounted on the tablet PC 100, and the user is a viewer on the road around the road surface. Is drawn using the tablet PC 100 to display a desired still image that can be visually recognized when viewing a still image projected on the road surface by the projector 2 attached to the display editing device, based on the still image drawn by the user It is assumed that graphic information for instructing the projector 2 to project is edited.

FIG. 10 is a block diagram showing a configuration example of the display editing device 1b according to the third embodiment.
In FIG. 10, the same components as those described with reference to FIG. 2 in the first embodiment are denoted by the same reference numerals, and redundant description will be omitted.
The display editing apparatus 1b differs from the display editing apparatus 1 according to the first embodiment in that the display editing apparatus 1b further includes a status information acquisition unit 121 and a graphic information acquisition unit 122.

The configuration of the display editing system according to the third embodiment is the same as the configuration described with reference to FIG. 1 in the first embodiment, and therefore redundant description will be omitted. Further, the configuration of the projector 2 according to the third embodiment is the same as the configuration described with reference to FIG. 3 in the first embodiment, and thus the redundant description will be omitted.

The status information acquisition unit 121 acquires status information of the projector 2. In the third embodiment, the situation information acquisition unit 121 acquires vehicle information on the vehicle as the situation information of the projector 2 from the car to which the projector 2 is attached. The vehicle information is various information related to the vehicle including the information related to the projector 2 attached to the vehicle, and is any information that can be a trigger for changing the still image projected from the projector 2. For example, the vehicle information may be position information of the car, position information of the projector 2 attached to the car, a state in which the car is parked, a direction in which the car is traveling, or a traveling speed of the car Information on the driving condition of the car.
The situation information acquisition unit 121 outputs the acquired vehicle information to the graphic information acquisition unit 122.

The graphic information acquisition unit 122 stores the situation of the projector 2 indicated by the vehicle information output from the situation information acquisition unit 121 in association with the second graphic information calculated in the situation in the storage unit 119. Therefore, in addition to the information described in the first and second embodiments, the storage unit 119 stores the situation of the projector 2 and the second graphic information corresponding to the situation in association with each other.
For example, assuming that the user is supposed to stay at one position where the viewer is supposed, a first figure to be viewed by the viewer at each of a plurality of different points with respect to the position. Are drawn on the screen of the tablet PC 100 to draw a plurality of first figures. The display editing apparatus 1b calculates second graphic information for each of a plurality of first graphic information indicating a plurality of first graphics. The graphic information acquisition unit 122 associates the position information of the related automobile with each of the plurality of pieces of second graphic information, and causes the storage unit 119 to store the positional information. Details will be described later.

The hardware configuration of the display editing apparatus 1b is the same as the configuration described with reference to FIGS. 6A and 6B in the first embodiment, and thus the redundant description will be omitted.
The status information acquisition unit 121 and the graphic information acquisition unit 122 include a captured image acquisition unit 111, a graphic reception unit 112, a positional relationship acquisition unit 113, a graphic position determination unit 114, a coordinate conversion unit 115, an instruction unit 116, and a display control unit. Similar to the audio output control unit 118 and the audio output control unit 118, the audio output control unit 118 can be realized by the processing circuit 601.

Next, the operation of the display editing device 1b of the third embodiment will be described.
In the following description, as an example, when the projector 2 changes the figure to be projected according to the position of the car, the user operates the display editing apparatus 1b to edit the figure at each position explain.
In this case, the user operates the touch panel of the tablet PC 100 in advance in a state where the vehicle to which the projector 2 is attached is at a plurality of different points, and causes the first displayed on the image displayed on the display unit 102. Draw a figure. Here, it is assumed that the plurality of points are, for example, three points, and the user draws a first figure for a state in which the car is at each of the three points.

The display editing apparatus 1b receives and determines the positional relationship between the road surface and the camera 101, and the positional relationship between the projector 2 and the camera 101, each time at the three points, accepting the first figure drawn by the user. From the positional relationship information and the first graphic information, second graphic information to be projected on the road surface by the projector 2 is calculated. The operation is the same as the operation described using the flowchart in FIG. 4 in the first embodiment, but in the third embodiment, the following operation is further performed in the display editing device 1b.

In the display editing apparatus 1b, when the coordinate conversion unit 115 calculates second graphic information (see step ST405 in FIG. 4), the coordinate conversion unit 115 outputs the second graphic information to the instruction unit 116 and the graphic information acquisition unit 122. The graphic information acquisition unit 122 stores the second graphic information in the storage unit 119 in association with the position of the projector 2 indicated by the vehicle information output from the situation information acquisition unit 121.
The timing at which the coordinate conversion unit 115 outputs the second graphic information to the graphic information acquisition unit 122 is at least one second graphic information calculated by the coordinate conversion unit 115 for one position of the projector 2. As long as it is stored. For example, when the user has finished drawing the first figure at one position of the projector 2, the user designates the timing when touching the storage button displayed on the display unit 102 or the like. be able to.
As a result, the position of the projector 2 and the second graphic information are associated with each other and stored in the storage unit 119 in each of three cases where the vehicle is at three different points in view of one assumed position of the viewer. It will be.

In the above description, the operation of the display editing apparatus 1b in the case where the vehicle information stored in the storage unit 119 in association with the second graphic information is used as the positional information of the projector 2 has been described. However, this is only an example, and the display / editing apparatus 1b can associate the second graphic information with the vehicle information other than the position information of the projector 2 and store it in the storage unit 119. Explain.

For example, the display editing apparatus 1b acquires, as the vehicle information, information on the parking state of the car or the direction in which the car travels, associates the second graphic information different from each other in the storage unit 119 with the states indicated by the respective information. It can be memorized.
The vehicle information in this case is information prepared for graphic editing, and the vehicle does not have to be in a state actually indicated by the vehicle information.

Further, for example, the display editing device 1b may obtain, as the vehicle information, information on the traveling speed of the car, and store the different pieces of second graphic information in the storage unit 119 in association with the states indicated by the respective information. it can.
The vehicle information in this case is also information prepared for graphic editing, and it is not necessary for the vehicle to actually travel at the traveling speed indicated by the vehicle information. Also, in this case, the user draws a first figure for each of a plurality of traveling speeds obtained by dividing the traveling speed, for example, every 10 km / hour, and the graphic information acquisition unit 122 generates the first graphic. With respect to the first graphic information indicating the graphic, the different second graphic information calculated by the coordinate conversion unit 115 may be stored in the storage unit 119 in association with the traveling speed.

Further, for example, the display editing device 1b may use the threshold of the traveling speed of the car, the second figure when the traveling speed of the car is less than the threshold, and the second figure when the traveling speed of the car becomes equal to or more than the threshold. The aspect of the modification may be stored in the storage unit 119. The mode of deformation of the second graphic can be, for example, enlargement of the second graphic, or blinking.

As described above, in the display editing device 1b, it is possible to appropriately set which second graphic is to be stored in association with the situation of the projector 2 indicated by which vehicle information.

Further, in the display editing device 1 b, the plurality of pieces of second graphic information stored in the storage unit 119 are associated with the second graphic information associated with one situation of the projector 2 and the other situations of the projector 2 It is possible to complement the existing second image information with the second image information to calculate the second image information used in an intermediate situation between each situation. For example, the position of the projector 2 and the second graphic information are associated with each other and stored in the storage unit 119 in each of three cases where the car is at three different points in view of one assumed position of the viewer. In this case, image processing is performed to continuously deform two pieces of second graphic information associated with two adjacent points among the three pieces of second graphic information already stored, and Second graphic information in the case where there is a car between can be calculated.
In this case, the projector 2 can project the continuously changing second graphic information on the road surface according to the movement of the automobile, and can perform graphic display without a sense of discomfort for the viewer.

As described above, according to the third embodiment, the second graphic information including the second graphic information and the second graphic information acquired by the state information acquiring unit 121 that acquires the state of the drawing apparatus is obtained. The storage unit 119 stores the information in association with the information indicating the state of the drawing apparatus when the information is calculated. Thereby, even when a change in the situation regarding the drawing device occurs, depending on the situation, the viewer can recognize a desired figure when looking at the figure drawn on the object from the non-facing position. The graphic information for instructing the drawing device to draw can be edited.

In the third embodiment described above, the third embodiment is applied to the first embodiment, and the display editing device 1 b is different from the display editing device 1 of the first embodiment in the situation information acquisition unit 121 and the figure. The information acquisition unit 122 is further provided. However, the present invention is not limited to this, and the third embodiment may be applied to the second embodiment. That is, the display editing apparatus 1 b may further include a status information acquisition unit 121 and a graphic information acquisition unit 122 in addition to the display editing apparatus 1 a of the second embodiment described with reference to FIG. 7.

Further, in the above first to third embodiments, as an example, the projector 2 attached to a car projects a figure as a still image on the road surface, and the display editing device 1 to 1b is a still image It is assumed that the figure is edited.
However, this is only an example, and when the projector 2 attached to a car projects a graphic as a moving image on a road surface, the display editing apparatus 1-1 b edits the graphic as a moving image It can also be done. In this case, for example, the user may draw an animation such as an animation as the first figure using an existing drawing application or the like.

In addition, the second graphic information calculated in the display editing apparatus 1-1 b can be extracted in any data format, regardless of whether the second graphic information is a still image or a moving image. Therefore, for example, when it is desired that the second graphic indicated by the second graphic information calculated based on the first graphic information is also projected from the projector attached to another car in the display editing device 1 to 1b. Takes out the second graphic information from the display editing device 1-1 b and copies the data to a storage device of a control device that controls a projector attached to another car, It can be projected and reproduced from a projector attached to a car of

In the first to third embodiments described above, the display editing devices 1 to 1b are mounted on the tablet PC 100, but the present invention is not limited to this. The device on which the display editing device 1-1 b is mounted can view the real space on which the second graphic is drawn on the display screen or directly by the user, and the user can Any device can be used as long as it can input and display a still image or a moving image so as to overlap and be seen in space. For example, a video camera is installed on the back of the display of a normal PC, and the user captures an image of the real space with the video camera and displays it on the display screen of the PC. The first figure may be drawn on the real space image. Also, for example, an apparatus on which the display editing apparatus 1 to 1b is mounted is a see-through head mounted display, and the user mounts the head mounted display and recognizes the user's fingertips with a three-dimensional tracker. The configuration may be such that the user can draw the first figure over the real space directly while looking at the real space.

In the first to third embodiments, the target on which the graphic is drawn is the road surface, and the drawing device is the projector 2 that projects light on the road surface. However, the present invention is not limited to this.
For example, the drawing apparatus may be a robot that draws on a whiteboard or the like by an arm holding a marker or the like, or may be a self-propelled robot that draws on a ground or the like using a tape or the like.
As described above, the object is not limited to the road surface, and may be an arbitrary surface such as a white board. Further, the object may be a surface including unevenness. Further, the drawing device is not limited to the projector 2 but may be any device capable of drawing a figure, such as an arm of a robot. Further, the drawing apparatus may be an apparatus in which a mask whose shape is mechanically changed is attached to a light source, and a figure is visually recognized by changing the shape of the mask.

In the above first to third embodiments, the drawing device is the projector 2 that projects light on the road surface, and the display editing devices 1 to 1b use the second graphic information for causing the projector 2 to project a desired graphic. It was to edit. In this case, the second figure projected from the projector 2 may be deformed on the road surface due to the positional relationship between the projector 2 and the road surface, etc. Therefore, the coordinate conversion unit 115 of the display editing device 1-1 b The second graphic information is calculated based on the information of the positional relationship between the camera 2 and the camera 101 and the positional relationship between the projector 2 and the camera 101.
However, for example, in the case of a robot in which a drawing device draws on a whiteboard or the like with an arm holding a marker or a self-propelled robot for drawing on the ground or the like using a tape or the like, Since the second graphic itself is obtained, the coordinate conversion unit 115 can calculate the second graphic information as long as the positional relationship between the camera 101 and the object is known.
Therefore, in this case, it is sufficient that the positional relationship acquisition unit 113 acquire the positional relationship between the camera 101 and the target object, and the coordinate conversion unit 115 calculates the positional relationship information between the camera 101 and the target object. The second graphic information may be calculated from the first graphic information.

Furthermore, within the scope of the present invention, the present invention allows free combination of each embodiment, or modification of any component of each embodiment, or omission of any component in each embodiment. .

In the display editing apparatus according to the present invention, graphic information for instructing the drawing apparatus to draw so that a desired figure can be visually recognized when the viewer looks at the figure drawn on the object from the non-facing position. Since editing is possible, the present invention can be applied to a display editing apparatus or the like for editing graphic information for instructing drawing to draw a figure on an arbitrary target object.

Reference Signs List 1 display editing apparatus, 2 projector, 51 car, 100 tablet PC, 101 camera, 102 display unit, 103 audio output device, 111 captured image acquisition unit, 112 graphic reception unit, 113 positional relationship acquisition unit, 114 graphic position determination unit, 115 coordinate conversion unit, 116 instruction unit, 117 display control unit, 118 audio output control unit, 119 storage unit, 120 figure complementation unit, 121 situation information acquisition unit, 122 graphic information acquisition unit.

Claims

A captured image acquisition unit that acquires an image obtained by capturing an object by a camera;
A graphic receiving unit that receives first graphic information on a first graphic drawn on the image;
A positional relationship acquisition unit that acquires a positional relationship between the object and the camera when the image is captured;
A coordinate conversion unit that calculates second graphic information from the positional relationship acquired by the positional relationship acquisition unit and the first graphic information;
An instruction unit that outputs the second graphic information calculated by the coordinate conversion unit to a drawing device that performs drawing on the object.
The drawing device is a projector for projecting light onto the object,
The positional relationship acquiring unit acquires the positional relationship between the drawing device and the object when the image is captured, in addition to the positional relationship between the object and the camera when the image is captured. And
The coordinate conversion unit is configured to obtain, from the positional relationship between the object and the camera, the positional relationship between the drawing device and the object, and the first graphic information, acquired by the positional relationship acquisition unit. The display editing apparatus according to claim 1, wherein the second graphic information is calculated.
The display editing apparatus according to claim 1, wherein the drawing apparatus is an apparatus for drawing directly on the object.
The first figure is complemented so that the shape of the first figure on the object is symmetrical based on the first figure information accepted by the figure accepting unit, and the complemented first figure information is Equipped with a figure complementing unit to create
The coordinate conversion unit is configured to calculate the second graphic information from the positional relationship acquired by the positional relationship acquisition unit and the post-completion first graphic information generated by the graphic complement unit. The display editing apparatus according to 1).
A situation information acquisition unit for acquiring the situation of the drawing apparatus;
Graphic information to be stored in the storage unit by associating the second graphic information with the information indicating the status of the drawing apparatus when the second graphic information is calculated, acquired by the status information acquisition unit The display editing apparatus according to claim 1, further comprising: an acquisition unit.
A portable device comprising: the display editing apparatus according to claim 1; the camera; and a touch panel for displaying an image obtained by imaging the object and drawing the first figure.
The first figure is complemented so that the shape of the first figure on the object is symmetrical based on the first figure information accepted by the figure accepting unit, and the complemented first figure information is Equipped with a figure complementing unit to create
The coordinate conversion unit calculates the second graphic information from the positional relationship acquired by the positional relationship acquisition unit and the post-completion first graphic information generated by the graphic complement unit.
The figure complementing unit
The display editing apparatus according to claim 6, wherein a guide for making the first figure symmetrical in left and right is displayed on the touch panel.
A captured image acquisition unit acquires an image obtained by imaging a target object by a camera;
The figure accepting unit accepts first figure information on a first figure drawn on the image;
A positional relationship acquisition unit acquiring a positional relationship between the object and the camera when the image is captured;
A step of calculating second graphic information from the positional relationship acquired by the positional relationship acquisition unit and the first graphic information;
And D. outputting the second graphic information calculated by the coordinate conversion unit to a drawing device for drawing on the object.
On the computer
The camera captures an image of the object,
Accepting first graphic information on a first graphic drawn on the image;
Acquiring a positional relationship between the object and the camera when the image is captured;
Second graphic information is calculated from the acquired positional relationship and the first graphic information;
A display editing program for executing a process of outputting the calculated second graphic information to a drawing device for drawing on the object.