JP2018160835A - Control method for display unit and display unit - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a control method for display unit and display unit, capable of suppressing discomfort from being given to an occupant of a mobile body.SOLUTION: The control method for display unit includes: projecting a picked-up image in a three-dimensional shape surrounding a circumference of a mobile body; creating a video with the mobile body and the circumference of the mobile body viewed from a virtual viewpoint positioned outside the three-dimensional shape; identifying an imaging direction of a camera and a region of a projection surface which the virtual viewpoint faces; detecting whether or not an object exists in the region of the projection surface; identifying a direction of an object relative to the camera when the object exists; and displaying a video in which the direction of the object is displayed in an opposite direction.SELECTED DRAWING: Figure 9

Description

  The present invention relates to a display device control method and a display device.

  With reference to one or a plurality of cameras, spatial reconstruction means for mapping an input image from the camera to a predetermined spatial model of a predetermined three-dimensional space, and spatial data mapped by the spatial reconstruction means, 2. Description of the Related Art An image generation apparatus including a viewpoint conversion unit that creates an image viewed from an arbitrary virtual viewpoint in a predetermined three-dimensional space and a display unit that displays an image converted by the viewpoint conversion unit is known (patent) Reference 1). The image generation apparatus is mounted on a moving body and displays an image of the moving body and the surroundings of the moving body viewed from a virtual viewpoint set at a position where the moving body is looked down obliquely downward.

International Publication No. 2000-07373

  In the prior art, a captured image of a mirror-inverted camera is displayed in an area of a projection plane in which the direction of the virtual viewpoint and the optical axis direction of the camera face each other in the image viewed from the virtual viewpoint. For this reason, when the occupant of the moving body sees the display image in which the mirror is inverted, there is a problem that the occupant of the moving body feels uncomfortable.

  The present invention is to suppress giving a sense of incongruity to a passenger of a moving body.

  The present invention projects a captured image onto a three-dimensional shape that surrounds the periphery of the moving object, generates an image of the moving object and the surroundings of the moving object from a virtual viewpoint located outside the three-dimensional shape, Identify the area of the projection plane that the virtual viewpoint is facing, detect whether there is an object within the area of the identified projection plane, and identify the direction of the object relative to the camera when the object exists. The problem is solved by displaying the direction of the object in the opposite direction.

  ADVANTAGE OF THE INVENTION According to this invention, it can suppress giving a discomfort to the passenger | crew of a moving body.

FIG. 1 is a configuration diagram of a display system including a display device according to the first embodiment. FIG. 2 is a diagram illustrating an example of an installation position of a camera mounted on the host vehicle. FIG. 3 is a diagram for explaining a panoramic image and a virtual viewpoint. FIG. 4 is a diagram for explaining the target area and the virtual viewpoint video. FIG. 5 is a diagram illustrating an example of a pedestrian icon image. FIG. 6 is a diagram illustrating an example of a scene where it is necessary to grasp the situation around the host vehicle. FIG. 7 is a diagram illustrating an example of a virtual viewpoint video displayed by the display device according to the comparative example. FIG. 8 is a diagram illustrating an example of a virtual viewpoint video displayed by the display device according to the first embodiment. FIG. 9 is a flowchart showing a control procedure of the display device according to the first embodiment. FIG. 10 is a diagram illustrating an example of the other vehicle icon image and the two-wheeled vehicle icon image. FIG. 11 is a diagram illustrating another example of a scene where it is necessary to grasp the situation around the host vehicle. FIG. 12 is a diagram illustrating an example of a virtual viewpoint video displayed by the display device according to the second embodiment. FIG. 13 is a diagram illustrating an example of a virtual viewpoint video displayed by the display device according to the third embodiment. FIG. 14A is a diagram illustrating an example of a virtual viewpoint video displayed by the display device according to the fourth embodiment. FIG. 14B is a diagram illustrating another example of the virtual viewpoint video displayed by the display device according to the fourth embodiment. FIG. 15 is a diagram illustrating an example of a display processing process for processing a pedestrian display.

<< First Embodiment >>
Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the present embodiment, a case where the display device according to the present invention is applied to a display system 1 mounted on a moving body will be described as an example. The display system 1 of the present embodiment displays a video for grasping the moving body and the surroundings of the moving body on a display viewed by an operator of the moving body.

  FIG. 1 is a block configuration diagram of a display system 1 including a display device 100 according to the present embodiment. As shown in FIG. 1, the display system 1 of this embodiment includes a display device 100 and a mobile device 200. Each device of the display device 100 and the mobile device 200 includes a wired or wireless communication device (not shown), and exchanges information with each other.

  The moving body to which the display system 1 of the present embodiment is applied includes a vehicle, a submarine explorer, an armored vehicle, a train, a forklift, and other devices having a moving function. In the present embodiment, a case where the moving body is a vehicle will be described as an example. Note that the moving body of the present embodiment may be a manned machine on which a human can be boarded, or an unmanned machine on which a human is not boarding.

  Note that the display system 1 of the present embodiment may be configured as an apparatus mounted on a moving body, or may be configured as a portable apparatus that can be brought into the moving body. In addition, a part of the configuration of the display system 1 according to the present embodiment may be mounted on a moving body, and another configuration may be mounted on a device physically different from the moving body, and the configuration may be distributed. In this case, the mobile body and another device are configured to be able to exchange information.

  As shown in FIG. 1, the mobile device 200 of the present embodiment includes a camera 20, a controller 30, an occupant operation device 40, and a display 50. Each of these devices is connected by a LAN (CAN) (Controller Area Network) or other LAN mounted on a mobile body, and can exchange information with each other.

  The camera 20 of the present embodiment is provided at a predetermined position of a vehicle (an example of a moving body, the same applies hereinafter). The number of cameras 20 and the installation positions of the cameras 20 are not particularly limited, but are preferably the number and installation positions necessary for imaging the surroundings of the vehicle. The camera 20 mounted on the vehicle images the vehicle and / or the surroundings of the vehicle and sends the captured image to the display device 100. The captured image in the present embodiment includes a part of the vehicle and a video around the vehicle. The captured image data is used for generation processing of a vehicle or a video around the vehicle and detection processing of an object from the generated video.

  FIG. 2 is a diagram illustrating an example of an installation position of the camera 20 mounted on the host vehicle V. 2A is a front view of the host vehicle V, FIG. 2B is a left side view of the host vehicle V, and FIG. 2C is a top view of the host vehicle V. FIG. As shown in FIGS. 2A to 2C, the host vehicle V has a front camera 21 installed in front of the host vehicle V and a rear camera 22 installed in the rear of the host vehicle V. 21 and the rear camera 22 face each other. The host vehicle V has a left camera 23 installed on the left side of the host vehicle V and a right camera 24 installed on the right side of the host vehicle V. The left camera 23 and the right camera 24 face each other. ing. In the example shown in the figure, the front camera 21 is installed on the bumper of the host vehicle V, the rear camera 22 is installed around the rear window of the host vehicle V, the left camera 23 is installed on the right door mirror, and the right camera 24 is installed. Is installed in the right door mirror. In addition, the installation position of the camera 20 is not specifically limited, The installation position can be changed arbitrarily.

  Moreover, the camera 20 has a wide-angle lens and images an object around the vehicle. Examples of the camera 20 include a CCD camera and a CMOS camera. In the example shown in the figure, the front camera 21 images the front of the host vehicle V, the rear camera 22 images the rear of the host vehicle V, the left camera 23 images the left side of the host vehicle V, and the right camera. 24 images the right side of the host vehicle V. Note that the direction in which the camera 20 captures an image (the capturing direction of the camera 20) is not particularly limited, and the capturing direction can be arbitrarily set. Each camera 20 sends a captured image to the display device 100 at a command from the control device 10 described later or at a preset timing.

  The imaging direction of the camera 20 is determined according to the optical axis direction of the camera 20. In the present embodiment, the optical axis direction of the camera 20 is a direction in which the lens of the camera 20 faces. The optical axis is a straight line that passes through the center of the lens and is perpendicular to the lens surface. The imaging direction of the camera 20 is used to specify the area of the projection plane that faces the virtual viewpoint among the projection planes described later.

  In the present embodiment, the captured image captured by the camera 20 is used for generating a video and for detecting an object. The captured image of the camera 20 of the present embodiment includes an image of an object around the vehicle. The target object in the present embodiment includes a ground surface around the moving body and an object around the moving body.

  In the present embodiment, the ground surface is a term indicating a concept including the surface of the earth and the surface of the earth's crust (land). The target object includes other vehicles, pedestrians, road structures, parking lots, signs, facilities, and other objects existing around the host vehicle V.

  The controller 30 of this embodiment controls the operation of the moving object including the host vehicle V. The controller 30 centrally manages the captured image data captured by the mobile camera 20.

  The occupant operation device 40 of the present embodiment is a display switch that is activated by the display device 100 and is operated by an occupant of the host vehicle V. When the display switch is turned on, the display device 100 is activated and displays an image on the display 50. Conversely, when the switch is turned off, the display device 100 stops and does not display an image on the display 50. The occupant operation device 40 may include a configuration for selecting a three-dimensional shape for projecting a captured image, which will be described later, and a configuration for setting a virtual viewpoint.

  The display 50 according to the present embodiment displays an image of the host vehicle V and the surroundings of the host vehicle V viewed from an arbitrary virtual viewpoint generated by the display device 100 described later. In this embodiment, the display system 1 in which the display 50 is mounted on a moving body will be described as an example. However, the display 50 may be provided on the portable display device 100 side that can be brought into the moving body.

  Next, the display device 100 of this embodiment will be described. The display device 100 of this embodiment includes a control device 10.

  As shown in FIG. 1, the control device 10 of the display device 100 according to the present embodiment stores a ROM (Read Only Memory) 12 in which a program for displaying a moving body and surrounding images is stored, and the ROM 12 stores the program. By executing the program, a CPU (Central Processing Unit) 11 as an operation circuit for realizing the function of the display device 100 and a RAM (Random Access Memory) 13 functioning as an accessible storage device are provided. An MPU (Micro Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like may be used as the operation circuit. The control device 10 may include a GPU (Graphics Processing Unit) that executes image processing.

  The control device 10 of the display device 100 according to the present embodiment includes an image acquisition function, a three-dimensional shape setting function, a panoramic video generation function, a virtual viewpoint video creation function, a target area specifying function, a target object detection function, An object display change function and a display control function are realized. The control apparatus 10 of this embodiment performs each function by cooperation of the software for implement | achieving the said function, and the hardware mentioned above. In the following description, it is assumed that the occupant of the host vehicle V operates the occupant operating device 40 and the display switch is on.

  Hereinafter, each function realized by the control device 10 will be described.

  First, the image acquisition function of the control device 10 will be described. The control device 10 according to the present embodiment acquires captured image data captured by the camera 20. The display device 100 acquires captured image data from the mobile device 200 using a communication device (not shown).

  Next, the three-dimensional shape setting function of the control device 10 will be described. A solid shape (space model) defined in advance for projecting a captured image is stored in the ROM 12. The control device 10 sets the space model by reading the space model from the ROM 12. Note that the method of setting the space model is not limited to the method of reading from the ROM 12. For example, a method in which an occupant of the host vehicle V inputs via the occupant operation device 40 or a method in which an occupant of the host vehicle V selects from a plurality of types of spatial models prepared in advance via the occupant operation device 40 may be used. . As will be described in detail later, the space model is a three-dimensional space representing a three-dimensional space, and is used when projecting an image captured by the camera 20.

  The aspect of the spatial model is not particularly limited. Examples of the shape of the space model include a rectangular parallelepiped, a cube, a polygonal column, a cylinder, a sphere, and a hemisphere. The space model is for projecting a surrounding image when the host vehicle V (moving body) exists inside. For this reason, the host vehicle V exists inside the space model regardless of the shape of the space model. In other words, the space model surrounds the host vehicle V. The space model is composed of a coordinate system (address). In this embodiment, a case where a cylindrical space model is set will be described.

  Next, the panoramic video generation function of the control device 10 will be described with reference to FIG. FIG. 3 is a diagram for explaining a panoramic image and a virtual viewpoint.

  The control device 10 projects a captured image of the camera 20 onto a cylindrical space model P. The projection method is not particularly limited. The control device 10 may execute the projection process using a projection mapping technique known at the time of filing. The projection processing may be executed using correspondence information between the address of the captured image data and the address (coordinate value) of the space model P.

  The control device 10 generates panoramic video PI showing the surroundings of the host vehicle V by mapping the data of the captured image of the camera 20 to the space model P. The shape of the generated panoramic image PI corresponds to the shape of the space model P. When the space model P has a cylindrical shape, the control device 10 maps the captured image data of the camera 20 onto the side surface S of the space model P, thereby generating a cylindrical panoramic image PI. Note that the control device 10 can generate the panoramic video PI by appropriately changing the mapping position of the captured image data of the camera 20 according to the shape of the space model P. For example, when the space model P is a rectangle, the control device 10 maps the data of the captured image of the camera 20 on the side surface of the rectangle.

In addition, the control device 10 displays the host vehicle V inside the space model P. The display method of the own vehicle V in the space model P is not particularly limited. As a display of the host vehicle V, a captured image of the host vehicle V captured in advance may be displayed superimposed on the space model P. Further, a vehicle icon image V ^′ indicating the host vehicle V prepared in advance may be displayed superimposed on the space model P. The vehicle icon image V ^′ may be created in advance based on the design of the host vehicle V and stored in the ROM 12. In this way, by superimposing the vehicle icon image V ^′ on the video, the relationship between the position and orientation of the host vehicle V and the surrounding video can be shown in an easily understandable manner. In the present embodiment, the display of the host vehicle V in the space model P will be described using a vehicle icon image V ^′ .

The control device 10 determines the position of the vehicle icon image V ^′ inside the space model P and determines the direction of the vehicle icon image V ^′ . In the present embodiment, the control device 10 treats the cylindrical bottom surface B as a road surface on which the host vehicle V travels, and displays a vehicle icon image V ^′ at the center of the bottom surface B. The vehicle icon image V ^'position of is not particularly limited, the vehicle icon image ^V' the position of may be changed after setting or set at an arbitrary position.

The control device 10 generates a surrounding image (panoramic image PI) of the host vehicle V according to the shape of the space model P. Controller 10 ^'on the side surface of the front side of the cylindrical shape as viewed from mapping the captured image data of the front camera 21, the vehicle icon image ^V' vehicle icon image V on the side surface of the rear side of the cylindrical shape as viewed from, The captured image data of the rear camera 22 is mapped. Further, the control unit 10, the vehicle ^'on the side surface of the cylindrical when viewed from the left side, mapping the captured image data of the left camera 23, the vehicle icon image ^V' icon image V cylinder right side as seen from The captured image data of the right camera 24 is mapped to the side of the shape. As a result, a panoramic image PI indicating an image of the entire periphery of the host vehicle V is generated along the cylindrical arcuate side surface S. In FIG. 3, a cylindrical panoramic image PI is indicated by diagonal lines.

When the control device 10 performs the above-described processing on the space model P, the data of the space model P after processing is handled as panoramic video information. The panoramic video information includes information such as the aspect of the space model P, the panoramic video including data of the image captured by the camera 20, and the position and orientation of the vehicle icon image V ^′ .

  Next, the virtual viewpoint video creation function of the control device 10 will be described with reference to FIGS. The control device 10 sets the virtual viewpoint VP as the reference viewpoint for the projection processing of panoramic video information. Specifically, the control device 10 determines the position and orientation of the virtual viewpoint.

The control device 10 according to the present embodiment sets the position of the virtual viewpoint VP outside the space model P. In FIG. 3, the virtual viewpoint VP is shown outside the space model P and at a position where the vehicle icon image V ^′ is looked down diagonally downward. In the present embodiment, the host vehicle V viewed from the virtual viewpoint VP and an image showing the surroundings of the host vehicle V are used in a scene for grasping the host vehicle V and the situation around the host vehicle V. For example, when the host vehicle V is about to enter an intersection, an occupant of the host vehicle V needs to grasp the left and right situations of the host vehicle V at the same time. Therefore, in order to grasp the situation of objects (another vehicle, a pedestrian, a road structure, a sign, a facility, other objects) around the intersection, the control device 10 places the vehicle icon image V ^′ diagonally downward, for example. The position of the virtual viewpoint VP is determined at the position to look down.

Then, the control device 10 sets the direction in which the vehicle icon image V ^′ is viewed from the position of the virtual viewpoint as the direction of the virtual viewpoint VP. Note that the control device 10 may calculate the viewpoint angle of the virtual viewpoint VP instead of the direction of the virtual viewpoint VP or together with the direction of the virtual viewpoint VP.

  In the present embodiment, the virtual viewpoint VP is stored in advance in the ROM 12 together with the space model P. Note that the virtual viewpoint VP is not limited to being set in advance, and an occupant of the host vehicle V may arbitrarily set the virtual viewpoint VP (position, direction) via the occupant operation device 40. Further, the virtual viewpoint VP may be set to an arbitrary position or changed after the setting.

  The control device 10 refers to the panoramic video information and the position and direction of the virtual viewpoint, and creates a projection video in which the panoramic video is viewed from the virtual viewpoint as a virtual viewpoint video. For example, the control device 10 generates a projected video from the virtual viewpoint VP by performing coordinate conversion on the data of the captured image of the camera 20 included in the panoramic video information and combining the data.

  FIG. 4 is a diagram for explaining the target area and the virtual viewpoint video. 4 shows a virtual viewpoint video PV viewed from the virtual viewpoint VP shown in FIG. For convenience of explanation, a virtual viewpoint VP is displayed in FIG.

In this embodiment, since the virtual viewpoint VP is installed outside the space model P, the entire image of the space model P onto which the captured image is projected is displayed on the created virtual viewpoint video PV. When the space model P has a cylindrical shape, the virtual viewpoint image PV created by the control device 10 displays the cylindrical space model P viewed from the virtual viewpoint VP, and also displays a panoramic image along the side surface S of the cylinder shape. PI is displayed. In the virtual viewpoint video PV, a vehicle icon image V ^′ is displayed at the center of the cylindrical bottom surface B.

  Next, the target area specifying function of the control device 10 will be described. The control device 10 identifies, as the target area TA, the area of the projection plane where the imaging direction of the camera 20 of the host vehicle V and the virtual viewpoint VP are opposed to each other in the created virtual viewpoint video PV. First, the control device 10 determines whether or not there is a region on the projection plane where the imaging direction of the camera 20 of the host vehicle V and the virtual viewpoint VP are opposed to each other in the created virtual viewpoint video PV.

The control device 10 defines the imaging direction of the camera 20. The control device 10 stores in advance in the ROM 12 the optical axis direction of the camera 20 (front camera 21, rear camera 22, left camera 23, right camera 24) for the host vehicle V as an optical axis vector. The camera 20 has a viewing angle, and the control device 10 defines the imaging direction of the camera 20 that expands radially by extending this optical axis vector to the entire viewing angle range centering on the camera 20. In the present embodiment, the camera 20 includes four cameras, and the control device 10 defines four imaging directions corresponding to the respective cameras. The control device 10 reflects information regarding the imaging direction of the specified camera 20 on the vehicle icon image V ^′ , and associates the imaging direction of the camera 20 of the vehicle icon image V ^′ with the imaging direction of the camera 20 of the host vehicle V. . Thereby, the imaging direction of the camera 20 can be handled in the virtual viewpoint video PV.

The control device 10 refers to the set position (address) and direction of the virtual viewpoint and the coordinates (address) of the vehicle icon image V ^′ , and calculates the direction from the virtual viewpoint to the host vehicle V as a virtual viewpoint vector. Thereby, in the virtual viewpoint video PV, the direction of the virtual viewpoint can be handled.

  And the control apparatus 10 of this embodiment determines whether the imaging direction of the camera 20 and a virtual viewpoint vector oppose in the virtual viewpoint video PV. For example, if there is one or more positions on the panoramic image PI included in the virtual viewpoint video PV where the respective imaging direction of each camera and the virtual viewpoint vector are superimposed, the control device 10 determines the imaging direction of the camera 20 and the virtual direction. It is determined that there is an area of the projection plane that faces the viewpoint.

  If the control device 10 determines that there is a region of the projection plane where the imaging direction of the camera 20 and the virtual viewpoint are opposite, the control device 10 defines a range of the projection plane where the imaging direction of the camera 20 and the virtual viewpoint are opposite. For example, the control device 10 sets a predetermined range along the panoramic video PI around the position on the panoramic video PI where the respective imaging direction of each camera and the virtual viewpoint vector are superimposed. The predetermined range is not particularly limited, and the control device 10 can set the predetermined range to an arbitrary range. The control device 10 may be set to the same range as the range of the captured image of the camera 20 being projected, or may be set to any other range. In FIG. 4, the target area TA is indicated by diagonal lines. Further, in FIG. 4, a target area TA is shown as an area of the projection plane where the imaging direction of the rear camera 22 of the host vehicle V and the virtual viewpoint VP face each other. The method for specifying the target area TA is not limited to the above method, and other methods may be used.

  As described above, the control device 10 according to the present embodiment has a projection plane in which the imaging direction of the camera 20 and the virtual viewpoint face each other around the position where the vector of the imaging direction of the camera 20 and the vector of the virtual viewpoint overlap. The area is specified as the target area TA.

  In addition, when there is no position on the panoramic image PI included in the virtual viewpoint video PV where the respective imaging directions of the cameras overlap the virtual viewpoint vector, the control device 10 determines the imaging direction of the camera 20 and the virtual viewpoint. It is determined that there is no region of the projection plane that the VP faces.

  Next, the object detection function of the control device 10 will be described. The control device 10 detects whether or not an object exists in the specified target area TA. First, the control device 10 specifies a captured image of the camera 20 projected on the target area TA as a captured image of the target area TA. Next, the control device 10 extracts features such as an edge, a color, a shape, and a size from the captured image data of the target area TA, and sets the attributes of the target object included in the captured image of the target area TA from the extracted features. Identify. The control device 10 stores the characteristics of each target object in advance, and identifies the target object included in the captured image of the target area TA by pattern matching processing. For example, the control device 10 detects other vehicles, pedestrians, two-wheeled vehicles, and other objects existing around the host vehicle V included in the captured image of the target area TA. The method for detecting the presence of the object using the captured image data of the target area TA is not particularly limited, and the method known at the time of filing the present application can be appropriately used. In the present embodiment, a pedestrian will be described as an example, but the object is not limited to a pedestrian.

  The feature point that the control device 10 extracts from the captured image data of the target area TA will be described. When specifying the pedestrian, the control device 10 walks the silhouette of the pedestrian's whole body (upper body and lower body), pedestrian clothes (tie, buttons, clothing color, etc.) from the captured image data of the target area TA. The head of the person, the face of the pedestrian (eyes, nose, mouth, hair color, etc.) are extracted. In addition, the feature point which the control apparatus 10 extracts is not restricted to said item, The control apparatus 10 may extract the other feature point of a pedestrian.

  In addition, when the feature point of the target object is extracted from the captured image data of the target area TA, the control device 10 specifies the direction of the target object with respect to the camera 20 from the extracted feature point of the target object. For example, when detecting the pedestrian from the captured image data of the target area TA, the control device 10 specifies the orientation of the pedestrian with respect to the camera 20 from the information regarding the pedestrian's face extracted from the captured image data of the target area TA. can do.

  The control device 10 specifies the direction of the object with respect to the camera 20 from the data of the captured image of the target area TA. There is no particular limitation on how to express the direction of the object. The control device 10 according to the present embodiment represents the direction of the object in four category expressions of front, back, left side, and right side. The control device 10 stores in advance representative examples of the four categories of each object as samples, and specifies a category corresponding to the orientation of the object with respect to the camera 20 by pattern matching processing. For example, if the control device 10 detects eyes, a nose, or the like as a part of a pedestrian's face viewed from the front, the control device 10 performs a pattern matching process and identifies the pedestrian's direction with respect to the camera 20 as the front. In addition, it is not limited to specifying a pedestrian's direction from one feature point of the pedestrian's feature points, You may specify a pedestrian's direction from the feature point of another pedestrian. Moreover, you may identify the direction of a pedestrian by combining these some feature points.

  Note that the control device 10 may define the direction of the object with respect to the camera 20 by an angle. The control device 10 can calculate the angle from the position (address) of the object and the position (address) of the virtual viewpoint VP. Thereby, the direction of the object with respect to the camera 20 can be detected with high accuracy.

Here, the orientation of the captured image of the camera 20 displayed in the target area TA in the virtual viewpoint video PV will be described. The control device 10 pastes (maps) the captured image of the camera 20 on the side surface S of the cylindrical space model P by the panoramic video generation function, and generates a panoramic video PI indicating the video around the host vehicle V. . On the other hand, the control device 10 provides a virtual viewpoint VP outside the cylindrical shape by a virtual viewpoint video creation function (see FIG. 3), and a virtual image obtained by viewing the panoramic video PI and the vehicle icon image V ^′ from the virtual viewpoint VP. A viewpoint video PV (see FIG. 4) is created. Since the vehicle icon image V ^′ is located at the center of the bottom surface of the cylindrical shape, the imaging direction of the camera 20 is a direction from the inside of the space model P to the outside of the space model P. On the other hand, since the virtual viewpoint VP is located outside the space model P, the direction of the virtual viewpoint VP is a direction from the outside of the space model P to the inside of the space model P. For example, when the space model P has a cylindrical shape, the imaging direction of the camera 20 is a direction that spreads radially from the center of the cylindrical shape to the side surface of the cylindrical shape. On the other hand, the direction of the virtual viewpoint is a direction overlooking the space model P from the outside of the cylindrical shape.

  The control device 10 creates a virtual viewpoint video PV in which the panoramic video PI is viewed from the virtual viewpoint VP even when the direction of the virtual viewpoint and the imaging direction of the camera 20 are opposite to each other. For this reason, in the virtual viewpoint video PV, the captured image of the camera 20 that is mirror-inverted is displayed in the area of the projection plane (target area TA) in which the imaging direction of the camera 20 and the orientation of the virtual viewpoint VP are opposite to each other.

  In this way, in the virtual viewpoint video PV, the captured image of the camera 20 that is mirror-reversed is displayed in the target area TA, so that the target object existing in the target area TA is opposite to the actual direction of the target object. Displayed in the direction. For this reason, when an occupant of the host vehicle V sees an object in the captured image of the camera 20 that is mirror-reversed, the occupant of the host vehicle V may feel uncomfortable. The control device 10 according to the present embodiment realizes a target area identification function, a target object detection function, and a target object display change function, thereby changing the orientation of the target object in the captured image of the camera 20 that has been mirror-inverted. A virtual viewpoint video PV in the same direction as the actual direction of the object is displayed. Thereby, it is suppressed that the passenger | crew of the own vehicle V gives discomfort.

  As described above, the control device 10 detects whether or not the target object exists in the target area TA, and when detecting the presence of the target object, the control device 10 determines the attribute of the target object, the direction of the target object with respect to the camera 20, and The feature points of the object are temporarily stored in the RAM 13.

  Next, the object display change function of the control device 10 will be described. The control device 10 displays the direction of the object relative to the camera 20 in the opposite direction with respect to the host vehicle V. The control device 10 reads information on the direction of the object with respect to the camera 20 from the RAM 13 and displays the direction of the object in the opposite direction.

  When the orientation of the object relative to the camera 20 is the front, the control device 10 changes the orientation of the object to the back and displays it. In addition, when the direction of the object with respect to the camera 20 is specified as the back surface, the control device 10 changes the direction of the object to the front surface and displays it. Further, when the direction of the object with respect to the camera 20 is specified as the left side, the control device 10 changes the direction of the object to the right side and when the direction of the object with respect to the camera 20 is specified as the right side. Displays the direction of the object changed to the left side.

  With reference to FIG. 5, a method of changing the display of the object will be described. FIG. 5 is a diagram illustrating an example of a pedestrian icon image. FIG. 5A is an example of a pedestrian icon image showing the front of the pedestrian, and FIG. 5B is an example of a pedestrian icon image showing the back of the pedestrian.

  When the target object exists in the target area TA, the control device 10 of the present embodiment displays the target object using an icon image corresponding to the attribute and orientation of the target object. An icon image corresponding to each object is stored in the ROM 12 in advance. For example, the ROM 12 stores a plurality of pedestrian icon images corresponding to the front, back, left side, and right side of the pedestrian. In addition, although this embodiment demonstrates the method of changing a pedestrian's display using an icon image, it is not limited to this.

  The control device 10 changes the display of the object by superimposing the display of the object in the captured image with an icon image. The icon image used by the control device 10 is an icon image in a direction opposite to the direction of the object with respect to the camera 20. Specifically, when the target object is a pedestrian, the control device 10 specifies a pedestrian icon image indicating the back of the pedestrian from the ROM 12 when the orientation of the pedestrian with respect to the camera 20 is identified as the front (see FIG. 5B). ). And the control apparatus 10 superimposes and displays the pedestrian icon image which shows the back of a pedestrian in the position (pixel) where the front of a pedestrian is displayed. Moreover, if the control apparatus 10 specifies the direction of the pedestrian with respect to the camera 20 as a back surface, the pedestrian icon image (refer FIG. 5 (A)) which shows the front of a pedestrian will be read from ROM12. And the control apparatus 10 superimposes and displays the pedestrian icon image which shows the front of a pedestrian in the position (pixel) where the back surface of a pedestrian is displayed.

  In the present embodiment, the case where the orientation of the pedestrian is specified as the front side or the back side is illustrated, but the control device 10 executes the above process even when the orientation of the pedestrian is specified as the left side surface or the right side surface. May be. That is, the control apparatus 10 will read the pedestrian icon image which shows the pedestrian's right side surface from ROM12, if the direction of a pedestrian is specified as a left side surface. And the control apparatus 10 may superimpose and display the pedestrian icon image which shows the pedestrian's right side in the position where the pedestrian's left side is displayed. Moreover, if the direction of a pedestrian is specified as a right side surface, the control apparatus 10 will read the pedestrian icon image which shows the left side surface of a pedestrian from ROM12. And the control apparatus 10 may superimpose and display the pedestrian icon image which shows the pedestrian's left side surface in the position where the pedestrian's right side surface is displayed.

  The pedestrian icon image is displayed while distinguishing at least the face, limbs, upper body, and lower body. Moreover, it is preferable to use the icon image which can distinguish a face and a body and can distinguish a front and a back as a pedestrian icon image. As in the example of FIG. 5A, eyes, nose, mouth, and the like may be displayed on the face of the pedestrian icon image, and the upper part of the pedestrian icon image has a clothing matching portion, a tie, Or a button etc. may be displayed. 5B, hair may be displayed on the back of the pedestrian icon image, and a buttock may be displayed on the lower half of the pedestrian icon image. By displaying using the pedestrian icon image in which the feature points on the front and back of the pedestrian are displayed, it is possible to accurately convey the direction of the pedestrian to the occupant of the host vehicle V through the virtual viewpoint video PV. it can.

  Thus, the control apparatus 10 displays the pedestrian direction with respect to the camera 20 in the opposite direction by superimposing the pedestrian icon image read from the ROM 12 on the captured image. Thereby, in the target area TA, the direction of the pedestrian viewed from the virtual viewpoint VP and the direction of the actual pedestrian with respect to the camera 20 can be made the same direction, and the passenger of the own vehicle V who sees the virtual viewpoint video PV feels uncomfortable. Can be suppressed.

  Note that the object display change function of the control device 10 performs display change processing for an object existing in the target area TA, but also changes the display for an object existing in an area other than the target area TA. You may perform the process of. For example, the control device 10 detects the target object from the captured image of the area other than the target area TA, and specifies the direction of the target object with respect to the camera 20. And the control apparatus 10 may read and display the icon image corresponding to the direction of the detected target object from ROM12. In this case, the objects present in the virtual viewpoint video PV are displayed in a unified manner as icon images, so that an occupant of the host vehicle V viewing the virtual viewpoint video PV can easily recognize the presence of the object. In addition, it is not limited to displaying all target objects as icon images, and captured images may be displayed as they are for target objects other than the target area TA.

  Next, the display control function of the control device 10 will be described. The control device 10 displays different virtual viewpoint videos PV on the display 50 based on the result of specifying the presence or absence of the target area TA or the result of detecting the presence or absence of the target. When the target area TA is not specified, the control device 10 determines that there is no mirror-inverted captured image in the virtual viewpoint video PV, and displays the virtual viewpoint video PV generated by the virtual viewpoint video creation function on the display 50. Display. Similarly, the control apparatus 10 displays the virtual viewpoint video PV generated by the virtual viewpoint video creation function on the display 50 when no target is detected in the target area TA. In this case, the control device 10 determines that the risk of giving a sense of discomfort to the occupant of the host vehicle V is low because the target object does not exist in the mirror-inverted captured image.

  On the contrary, when the target area TA is specified in the virtual viewpoint video PV and the control apparatus 10 detects that the target object is present in the target area TA, the pedestrian icon image corresponding to the actual direction of the pedestrian. Is superimposed on the captured image to create a virtual viewpoint video PV. The control device 10 displays the created virtual viewpoint video PV on the display 50. The display 50 may be mounted on the host vehicle V and configured as the mobile device 200 or may be provided on the display device 100 side. The display 50 may be a display for a two-dimensional image or a display that displays a three-dimensional image.

  Next, with reference to FIGS. 6 to 8, a virtual viewpoint image displayed on the display device 100 will be described using specific scenes. FIG. 6 is a diagram illustrating an example of a scene where it is necessary to grasp the situation around the host vehicle. FIG. 7 is a diagram illustrating an example of a virtual viewpoint video displayed by the display device according to the comparative example. FIG. 8 is a diagram illustrating an example of a virtual viewpoint video displayed by the display device 100 according to the present embodiment. The display device according to the comparative example is different in function from the control device 10 according to the present embodiment. The control device 10 according to the comparative example does not have the target area specifying function, the target object detection function, and the target object display change function as compared with the control device 10 according to the present embodiment.

  FIG. 6 shows a scene in which the host vehicle V1 enters from the road A1 with respect to an intersection of T-shaped roads with poor visibility, which is composed of the roads A1 to A3. In addition to the host vehicle V1, a pedestrian M1 enters from the road A2 and a pedestrian M2 enters the intersection from the road A3. The pedestrian M1 and the pedestrian M2 are located at opposite locations. The pedestrian M1 faces the front with respect to the imaging direction of the left camera 23 of the host vehicle V1, and the pedestrian M2 faces the front with respect to the imaging direction of the right camera 24 of the host vehicle V1. The pedestrian M1 is located within the imaging range PR3 of the left camera 23, and the pedestrian M2 is located within the imaging range PR4 of the right camera 24.

  In FIG. 6, the situation where the pedestrian M1 and the pedestrian M2 approach the own vehicle V1 is shown. Since it is difficult for the passengers of the host vehicle V1 to check the conditions of the roads A2 and A3, the host vehicle V1 checks whether there are pedestrians on the roads A2 and A3 before reaching the intersection. There is a need. The display device 100 and the display device according to the comparative example create a virtual viewpoint video in which the intersection is viewed from the virtual viewpoint VP1 in order to simultaneously grasp the situation of the roads on the left and right of the host vehicle V1 (road A2 and road A3). To the passenger of the host vehicle V1. In FIG. 6, for convenience of explanation, the virtual viewpoint VP1 is shown on the same plane as the host vehicle V1, the pedestrian M1, and the pedestrian M2, but actually, the virtual viewpoint VP1 is located at the left of the intersection. Located in a position looking down diagonally.

As illustrated in FIG. 7, the display device according to the comparative example creates a virtual viewpoint video in which the cylindrical space model P1 is viewed from the virtual viewpoint VP1 illustrated in FIG. 6. For convenience of explanation, FIG. 7 shows a virtual viewpoint VP1. The space model P1 is a panoramic image PI1 (the captured image C1 of the left camera 23 and the captured image of the right camera 24) mapped to the vehicle icon image V1 ^′ positioned at the center of the cylindrical bottom surface and the cylindrical side surface S1. C2). The captured image C1 includes a pedestrian image M1 ^′ showing the front of the pedestrian M1. The captured image C2 includes a pedestrian image M2 ^′ showing the front of the pedestrian M2.

As shown in FIG. 7, in the virtual viewpoint video PV1, the pedestrian image M1 ^′ is displayed with the front facing the virtual viewpoint VP1. This is because the captured image of the left camera 23 that is mirror-inverted is displayed in the region (target region) of the projection plane in which the imaging direction of the left camera 23 of the host vehicle V1 and the orientation of the virtual viewpoint VP1 face each other. is there. The actual pedestrian M1 facing the front with respect to the host vehicle V1 is displayed as the pedestrian image M1 ^′ facing the back with respect to the vehicle icon image V1 ^{′ in the} virtual viewpoint video PV1. Therefore, the occupant of the host vehicle V1 who sees the virtual viewpoint video PV1 cannot accurately grasp whether the pedestrian M1 is approaching or far from the host vehicle V1.

In the virtual viewpoint video PV1, the pedestrian image M2 ^′ is displayed with the front facing the virtual viewpoint VP1. This is because the imaging direction of the right camera 24 of the host vehicle V1 and the direction of the virtual viewpoint VP1 do not face each other, and the captured image of the right camera 24 is displayed as it is without being mirror-inverted. The actual pedestrian M2 facing the front with respect to the host vehicle V1 is displayed as the pedestrian image M2 ^′ facing the front with respect to the vehicle icon image V1 ^′ even in the virtual viewpoint video PV1. Therefore, the occupant of the host vehicle V1 viewing the virtual viewpoint video PV1 can accurately grasp that the pedestrian M2 is approaching the host vehicle V1.

On the other hand, as shown in FIG. 8, the display device 100 of the present embodiment creates a virtual viewpoint video PV1 ^′ obtained by viewing the cylindrical space model P1 from the virtual viewpoint VP1 shown in FIG. For convenience of explanation, FIG. 8 shows a virtual viewpoint VP1. A pedestrian icon image O1 indicating the back of the pedestrian M1 is superimposed on the captured image C1. The captured image C2 includes a pedestrian image M2 ^′ showing the front of the pedestrian M2. About the structure of the space model P1, since it is the structure similar to FIG. 7, the description is used.

As shown in FIG. 8, in the virtual viewpoint video PV1 ^′ , the pedestrian icon image O1 is displayed with the back facing the virtual viewpoint VP1. The control device 10 according to the present embodiment specifies the direction of the pedestrian M1 with respect to the camera 20 as the front from the captured image data of the target area. The control device 10 reads the pedestrian icon image O1 indicating the back of the pedestrian from the ROM 12, and displays it superimposed on the captured image C1. Thus, the actual pedestrian M1 facing the front with respect to the left camera 23 of the host vehicle V1 is displayed as the pedestrian icon image O1 facing the front with respect to the vehicle icon image V1 ^′ even in the virtual viewpoint video PV1 ^′. The

Note that the pedestrian image M2 ^′ is displayed with the front facing the virtual viewpoint VP1 in the virtual viewpoint video PV1 ^′ as well as the virtual viewpoint video PV1. That is, the actual pedestrian M2 facing the front with respect to the right camera 24 of the host vehicle V1 is displayed as the pedestrian image M2 ^′ facing the front with respect to the vehicle icon image V1 ^′ even in the virtual viewpoint video PV1 ^′. .

As described above, in the virtual viewpoint video PV1 ^′ displayed by the display device 100 according to the present embodiment, the pedestrian icon image O1 and the pedestrian image M2 ^′ viewed from the virtual viewpoint VP1 are viewed from the host vehicle V illustrated in FIG. It is displayed in the same direction as the actual pedestrians M1 and M2. As a result, the occupant of the host vehicle V1 can simultaneously check the left and right situations of the host vehicle V1, which are difficult to check from the host vehicle V1, through the virtual viewpoint video PV1 ^′ , and the direction of the object viewed from the host vehicle V1 can be accurately determined. Can grasp.

  Based on the flowchart of FIG. 9, operation | movement of the control apparatus 10 of this embodiment is demonstrated. FIG. 9 is a flowchart showing a control procedure of the display device according to the present embodiment. In step S <b> 101, the control device 10 detects the state of the display switch of the occupant operation device 40. If it is detected that the display switch is turned on, the process proceeds to step S102. If it is detected that the display switch is turned off, the process waits in step S101.

  In step S <b> 102, the control device 10 acquires a captured image captured by the camera 20 in order to acquire a surrounding image of the host vehicle V. When a plurality of cameras 20 are mounted on the host vehicle V, the control device 10 acquires captured images captured by the respective cameras. The acquired captured image is used for generating a video and for detecting an object.

  In step S103, the control device 10 reads the spatial model from the ROM 12 in order to set the three-dimensional shape (spatial model) for mapping the captured image acquired in step S102. Note that the shape of the space model may be changed during the operation of the display device 100. For example, when a plurality of types of space models are stored in the ROM 12, a configuration in which an occupant of the host vehicle V selects from the plurality of types of space models via the occupant operation device 40 while the host vehicle V is traveling may be used.

Then, the control device 10 displays the vehicle icon image V1 ^′ indicating the host vehicle V1 on the read space model, and maps the captured image acquired in step S102 to show a video around the host vehicle V. Generate panoramic video. For example, when the space model has a cylindrical shape, the control device 10 displays a vehicle icon image V1 ^′ indicating the host vehicle V on the cylindrical bottom surface. The control device 10 generates the cylindrical panoramic image by mapping the captured image data of the camera 20 on the cylindrical side surface.

  In step S104, the control device 10 sets a virtual viewpoint outside the space model set in step S103. The control device 10 sets the position of the virtual viewpoint and the direction of the virtual viewpoint. The virtual viewpoint is set to view the panoramic image generated in step S103 from an arbitrary position and orientation. For example, the control device 10 reads the virtual viewpoint stored in advance in the ROM 12.

  Then, the control device 10 creates a virtual viewpoint video in which the panoramic video is viewed from the virtual viewpoint. For example, when the spatial model is cylindrical, a cylindrical spatial model viewed from the virtual viewpoint VP is displayed on the virtual viewpoint video, and a panoramic video is displayed along the side surface of the cylindrical shape.

  In step S105, the control device 10 includes, in the virtual viewpoint video created in step S104, a projection plane area (target area) where the imaging direction of the camera 20 of the host vehicle V and the virtual viewpoint set in step S104 face each other. Judge whether to do. If it is determined that the target area exists, the process proceeds to step S106. If it is determined that the target area does not exist, the process proceeds to step S110.

  In step S106, the control device 10 detects whether or not an object is present in the target area. For example, the control device 10 extracts features such as edges from the data of the captured image of the camera 20, and specifies the attributes of the object from the extracted features. If a pedestrian, other vehicle, two-wheeled vehicle, or other object is detected as an object, the process proceeds to step S107. If no object is detected, the process proceeds to step S110.

  In step S <b> 107, the control device 10 specifies whether the direction of the object detected in step S <b> 106 is the front or back direction with respect to the camera 20. For example, the control device 10 specifies the orientation of the object with respect to the camera 20 from the features extracted from the captured image data in step S106. When the orientation of the object relative to the camera 20 is identified as the front, the process proceeds to step S108, and when the orientation of the object relative to the camera 20 is identified as the back, the process proceeds to step S109.

  In step S108, the control device 10 superimposes and displays an icon image indicating the back surface of the object at the position (pixel) where the front surface of the object is displayed. When the object is a pedestrian, the control device 10 reads a pedestrian icon image indicating the back of the pedestrian from the ROM 12, and superimposes the read pedestrian icon image on the position where the front of the pedestrian is displayed. To display.

  In step S109, the control device 10 superimposes and displays an icon image indicating the front of the object on the position (pixel) where the back of the object is displayed. When the object is a pedestrian, the control device 10 reads a pedestrian icon image indicating the front of the pedestrian from the ROM 12, and superimposes the read pedestrian icon image on the position where the back of the pedestrian is displayed. To display.

  In step S110, the virtual viewpoint video created in step S104 and the virtual viewpoint video in which the icon image of the target object is superimposed in step S108 or step S109 are displayed on the display 50.

  Finally, in step S111, the control device 10 detects the state of the display switch of the occupant operation device 40. When the on state of the display switch is detected, the process returns to step S102, and when the off state of the display switch is detected, the operation of the control device 10 is terminated.

  As described above, the control device 10 according to the present embodiment identifies the area of the projection plane where the imaging direction of the camera 20 and the virtual viewpoint face each other as the target area, and determines whether or not there is a pedestrian in the target area. To detect. Then, when there is a pedestrian, the control device 10 specifies the direction of the pedestrian with respect to the camera 20 and displays a virtual viewpoint image in which the direction of the pedestrian is displayed in the opposite direction on the display 50. Thereby, it can prevent showing the display of the pedestrian by which the mirror inversion was carried out to the passenger | crew of the own vehicle V, and it can suppress giving a discomfort to the passenger | crew of the own vehicle V. FIG.

  Furthermore, the control device 10 of the present embodiment displays the back of the pedestrian when the orientation of the pedestrian with respect to the camera 20 is the front, and displays the front of the pedestrian when the orientation of the pedestrian with respect to the camera 20 is the back. To do. Thereby, it is possible to accurately tell the occupant of the host vehicle V whether or not the pedestrian approaches the host vehicle V through the virtual viewpoint video.

  In addition, the control device 10 of the present embodiment displays the pedestrian's face and body orientation with respect to the camera 20 in the opposite direction using the pedestrian icon image indicating the pedestrian. Since each pedestrian icon image showing the front and back of the pedestrian shows the feature points of the front and back of the actual pedestrian, the direction of the pedestrian through the virtual viewpoint video to the passenger of the host vehicle V Can be conveyed accurately.

<< Second Embodiment >>
In the second embodiment, a case where another vehicle or a two-wheeled vehicle traveling around the host vehicle V is detected by using the display device according to the present invention will be described as an example. The display system of the present embodiment displays an image for grasping the moving body and the surroundings of the moving body on a display that is viewed by the operator of the moving body. In addition, the two-wheeled vehicle of this embodiment is a vehicle with two wheels, and the drive method is not limited. For example, bicycles include motorcycles and motorcycles.

  The display device according to the present embodiment has the same configuration as that of the display device according to the above-described embodiment except that the control device 110 (not shown) has different functions. Therefore, FIG. 1 is used in the above-described embodiment. Incorporated the explanation that was. The control device 110 according to the present embodiment is different from the control device 10 according to the above-described embodiment in an object detection function and an object display change function.

  Next, the object detection function of the control device 110 will be described. The control device 110 according to the present embodiment detects whether there is another vehicle or a two-wheeled vehicle in the target area. In addition, the detection method of presence of another vehicle or a two-wheeled vehicle specifies the attribute of the target object contained in the captured image of a target area | region by the method similar to the control apparatus 10 of embodiment mentioned above. The control device 110 according to the present embodiment extracts features such as edges, colors, shapes, and sizes from the captured image data of the target region, and identifies other vehicles or two-wheeled vehicles included in the captured image of the target region from the extracted features. To do.

  The feature points that the control device 110 extracts from the captured image data of the target area will be described. When specifying the other vehicle, the control device 110 determines the silhouette of the other vehicle, the color of the brake lamp or direction indicator of the other vehicle, the front window of the other vehicle, the headlight of the other vehicle from the captured image data of the target area. Extract the license plate of the other vehicle, the exhaust port of the other vehicle, and the rear window of the other vehicle. Further, when specifying the two-wheeled vehicle, the control device 110 determines the two-wheeled vehicle silhouette, the two-wheeled vehicle brake lamp or the direction indicator color, the two-wheeled vehicle headlight, the two-wheeled vehicle license plate, the two-wheeled vehicle Extract exhaust vents, motorcycle occupant silhouette (upper body), motorcycle occupant clothing (clothing color, etc.), motorcycle occupant head, motorcycle occupant face (eye, nose, mouth, hair color) . The feature points extracted by the control device 110 are not limited to the above items, and the control device 110 may extract other feature points of other vehicles or two-wheeled vehicles.

  In addition, when the feature point of the target object is extracted from the captured image data of the target area, the control device 110 specifies the direction of the target object with respect to the camera 20 from the extracted feature point of the target object. For example, when detecting the other vehicle from the captured image data of the target area, the control device 110 specifies the direction of the other vehicle with respect to the camera 20 from the information regarding the silhouette of the other vehicle extracted from the captured image data of the target area. Can do. Further, for example, when the two-wheeled vehicle is detected from the captured image data of the target region, the control device 110 may specify the direction of the two-wheeled vehicle with respect to the camera 20 from the information regarding the occupant of the two-wheeled vehicle extracted from the captured image data of the target region. it can.

  Similar to the control device 10 of the above-described embodiment, the control device 110 defines the detected orientation of the other vehicle or the two-wheeled vehicle with four category expressions of front, back, left side, and right side. The control device 110 stores in advance representative examples of the four categories of the other vehicle and the two-wheeled vehicle as samples, and specifies a category corresponding to the direction of the other vehicle or the two-wheeled vehicle with respect to the camera 20 by pattern matching processing. For example, when detecting the front window of another vehicle, control device 110 specifies the direction of the other vehicle relative to camera 20 as the front. Further, for example, when detecting the occupant (for example, eyes or nose) of a motorcycle viewed from the front, the control device 110 performs a pattern matching process and specifies the direction of the motorcycle with respect to the camera 20 as the front. The direction of the other vehicle or the two-wheeled vehicle is not limited to specifying the direction of the other vehicle or the two-wheeled vehicle from one of the characteristic points of the other vehicle or the two-wheeled vehicle, and the direction of the other vehicle or the two-wheeled vehicle is specified from the feature point of the other other vehicle or the two-wheeled vehicle. May be. Moreover, you may identify the direction of another vehicle or a two-wheeled vehicle combining these some feature points.

  As described above, the control device 110 detects whether or not there is another vehicle or two-wheeled vehicle in the target area, and when detecting the presence of the other vehicle or two-wheeled vehicle, the attribute of the other vehicle or two-wheeled vehicle and the other to the camera 20 are detected. The direction of the vehicle or the two-wheeled vehicle and the characteristic points of the other vehicle or the two-wheeled vehicle are temporarily stored in the RAM 13.

  Next, the object display change function of the control device 110 will be described. The control device 110 displays the direction of the other vehicle or the motorcycle with respect to the camera 20 in the opposite direction. Specifically, the control device 110 reads information on the direction of the object with respect to the camera 20 from the RAM 13 and displays the direction of the other vehicle or the motorcycle in the opposite direction.

  When the direction of the other vehicle or the two-wheeled vehicle with respect to the camera 20 is the front, the control device 110 changes the direction of the other vehicle or the two-wheeled vehicle to the back and displays it. In addition, when the direction of the other vehicle or the two-wheeled vehicle with respect to the camera 20 is the rear surface, the control device 110 changes the direction of the other vehicle or the two-wheeled vehicle to the front and displays it. Further, when the direction of the other vehicle or the two-wheeled vehicle with respect to the camera 20 is the left side, the control device 10 changes the direction of the other vehicle or the two-wheeled vehicle to the right side and displays the direction. In this case, the direction of the other vehicle or the motorcycle is changed to the left side and displayed.

  With reference to FIG. 10, a method of changing the display of the other vehicle and the motorcycle will be described. FIG. 10 is a diagram illustrating an example of the other vehicle icon image and the two-wheeled vehicle icon image. FIG. 10A is an example of another vehicle icon image showing the front of the other vehicle, and FIG. 10B is an example of other vehicle icon image showing the back of the other vehicle. FIG. 10C is an example of a two-wheeled vehicle icon image showing the front of the two-wheeled vehicle, and FIG. 10D is an example of a two-wheeled vehicle icon image showing the rear of the two-wheeled vehicle.

  Similarly to the control device 10 of the above-described embodiment, when the other vehicle or the two-wheeled vehicle exists in the target area TA, the control device 110 of the present embodiment displays the other vehicle or the two-wheeled vehicle with an icon image. For example, the ROM 12 stores a plurality of other vehicle icon images and a plurality of two-wheeled vehicle icon images respectively corresponding to the front, back, left side, and right side of other vehicles and two-wheeled vehicles. In addition, although this embodiment demonstrates the method to change the display of another vehicle or a two-wheeled vehicle using an icon image, it is not limited to this.

  The control device 110 changes the display of the other vehicle or the two-wheeled vehicle by superimposing and displaying the display of the other vehicle or the two-wheeled vehicle in the captured image with an icon image. The icon image used by the control device 110 is an icon image in the direction opposite to the direction of the other vehicle or the two-wheeled vehicle with respect to the camera 20.

  Specifically, when the object is an other vehicle and the direction of the other vehicle with respect to the camera 20 is identified as the front, the control device 110 reads from the ROM 12 an other vehicle icon image indicating the back of the other vehicle. And the control apparatus 110 superimposes and displays the other vehicle icon image which shows the back surface of another vehicle on the position (pixel) where the front of the other vehicle is displayed. Further, when the direction of the other vehicle with respect to the camera 20 is identified as the back surface, the control device 110 reads the other vehicle icon image indicating the front surface of the other vehicle from the ROM 12. And the control apparatus 110 superimposes and displays the other vehicle icon image which shows the front of another vehicle on the position (pixel) where the back surface of the other vehicle is displayed.

  In this embodiment, the case where the direction of the other vehicle is specified as the front side or the back side is illustrated. However, even when the direction of the other vehicle is specified as the left side surface or the right side surface, the control device 10 executes the above processing. May be. That is, when the direction of the other vehicle is specified as the left side surface, the control device 110 reads the other vehicle icon image indicating the right side surface of the other vehicle from the ROM 12. And the control apparatus 110 may superimpose and display the other vehicle icon image which shows the right side of the other vehicle in the position where the left side surface of the other vehicle is displayed. Moreover, if the direction of the other vehicle is specified as the right side surface, the control device 110 reads the other vehicle icon image indicating the left side surface of the other vehicle from the ROM 12. And the control apparatus 110 may superimpose and display the other vehicle icon image which shows the left side surface of the other vehicle in the position where the right side surface of the other vehicle is displayed.

  The other vehicle icon image distinguishes at least the brake lamp or direction indicator of the other vehicle, the front window of the other vehicle, the headlight of the other vehicle, the license plate of the other vehicle, the exhaust port of the other vehicle, and the rear window of the other vehicle. Displayed.

  Moreover, it is preferable to use the icon image which can distinguish a front and the back as an other vehicle icon image. As in the example of FIG. 10A, a headlight and a front window having a larger area than the rear window may be displayed on the other vehicle icon image indicating the front of the other vehicle. In addition, as shown in the example of FIG. 10B, the other vehicle icon image indicating the back of the other vehicle displays a brake lamp, a direction indicator, a license plate, an exhaust port, and a rear window whose area is smaller than that of the front window. May be. By displaying using the other vehicle icon image showing the front and back feature points of such other vehicles, the direction of the other vehicle can be accurately communicated to the passenger of the own vehicle V through the virtual viewpoint video PV. it can.

  Next, the motorcycle icon image will be described. In the present embodiment, in the two-wheeled vehicle icon image indicating the front of the two-wheeled vehicle, the front of the two-wheeled vehicle occupant is displayed instead of the front of the two-wheeled vehicle itself. Similarly, in the two-wheeled vehicle icon image indicating the rear surface of the two-wheeled vehicle, the rear surface of the occupant of the two-wheeled vehicle is displayed instead of the rear surface of the two-wheeled vehicle itself. In addition, it is not limited to distinguishing a front and a back by the passenger | crew of a two-wheeled vehicle, You may distinguish and display the front and back of a two-wheeled vehicle, respectively.

  When the target object is a two-wheeled vehicle, the control device 110 reads a two-wheeled vehicle icon image indicating the back of the occupant of the two-wheeled vehicle from the ROM 12 when the direction of the two-wheeled vehicle with respect to the camera 20 is specified as the front. And the control apparatus 110 superimposes and displays the other vehicle icon image which shows the back of the passenger | crew of a two-wheeled vehicle at the position (pixel) where the passenger | crew's front of a two-wheeled vehicle is displayed. Further, when the direction of the two-wheeled vehicle with respect to the camera 20 is specified as the back, the control device 110 reads a two-wheeled vehicle icon image indicating the front of the two-wheeled vehicle from the ROM 12. And the control apparatus 110 superimposes and displays the two-wheeled vehicle icon image which shows the front of the passenger | crew of a two-wheeled vehicle at the position (pixel) where the back of the passenger | crew of a two-wheeled vehicle is displayed.

  Like the example of FIG.10 (C), in the two-wheeled vehicle icon image which shows the front of a two-wheeled vehicle, in order to show a passenger | crew's front, you may display the button etc. of clothes. Further, as in the example of FIG. 10D, the two-wheeled vehicle icon image indicating the back of the two-wheeled vehicle may display the hair and the buttocks to indicate the back of the occupant. By displaying such a two-wheeled vehicle using the two-wheeled vehicle icon image showing the front and rear feature points, the direction of the two-wheeled vehicle can be accurately conveyed to the occupant of the host vehicle V through the virtual viewpoint video PV.

  As described above, the control device 110 displays the direction of the other vehicle or the two-wheeled vehicle with respect to the camera 20 in the opposite direction by superimposing the other vehicle icon image or the two-wheeled vehicle icon image read from the ROM 12 on the captured image. Thereby, in the target area, the direction of the other vehicle or the two-wheeled vehicle viewed from the virtual viewpoint VP and the direction of the actual other vehicle or the two-wheeled vehicle can be made the same direction, and the passenger of the own vehicle V viewing the virtual viewpoint video is uncomfortable. Giving can be suppressed.

  Next, with reference to FIGS. 11 and 12, a virtual viewpoint video displayed on the display device 100 will be described using specific scenes. FIG. 11 is a diagram illustrating another example of a scene where it is necessary to grasp the situation around the host vehicle. FIG. 12 is a diagram illustrating an example of a virtual viewpoint video displayed by the display device 100 according to the present embodiment.

  FIG. 11 shows a scene where the host vehicle V2 traveling on the road A4 changes lanes and joins the road A5 with respect to the junction of the road A4 and the road A5. The other vehicle V3 travels on the road A5 in front of the junction, and the other vehicle V4 travels on the road A5 behind the junction. The own vehicle V2 is changing the lane from this junction to the road A5 and is about to enter between the other vehicle V3 and the other vehicle V4. The other vehicle V3 faces the back with respect to the imaging direction of the front camera 21 of the host vehicle V2, and the other vehicle V4 faces the front with respect to the imaging direction of the rear camera 22 of the host vehicle V2. The other vehicle V3 is positioned within the imaging range PR1 of the front camera 21, and the other vehicle V4 is positioned within the imaging range PR2 of the rear camera 22.

  FIG. 11 shows a situation in which the other vehicle V3 moves away from the host vehicle V2 and the other vehicle V4 approaches the host vehicle V2. Since it is difficult for the passenger of the host vehicle V2 to check the state of the road A5, it is necessary to check whether there is another vehicle or a two-wheeled vehicle on the road A5 before the host vehicle V2 changes lanes. The display device 100 creates a virtual viewpoint video in which the junction point is viewed from the virtual viewpoint VP2 and displays it on the passenger of the host vehicle V2 in order to simultaneously grasp the situation before and after the road A5. In FIG. 11, for the sake of convenience of explanation, the virtual viewpoint VP2 is shown on the same plane as the host vehicle V2 and the other vehicles V3, V4. Located in a place overlooking.

As shown in FIG. 12, the display device 100 of the present embodiment creates a virtual viewpoint video PV2 ^′ obtained by viewing the cylindrical space model P2 from the virtual viewpoint VP2 shown in FIG. The space model P2 includes a vehicle icon image V2 ^' positioned at the center of the cylindrical bottom surface, a captured image C3 of the front camera 21 and a captured image C4 of the rear camera 22 mapped to the cylindrical side surface S2. The captured image C3 includes another vehicle image V3 ^′ that shows the back surface of the other vehicle V3. Moreover, the other vehicle icon image O4 which shows the back surface of the other vehicle V4 is superimposed on the captured image C4.

As shown in FIG. 12, in the virtual viewpoint video PV2 ^′ , the other vehicle icon image O4 is displayed with the back side facing the virtual viewpoint VP2. The control device 110 of the present embodiment specifies the direction of the other vehicle V4 with respect to the camera 20 as the front from the captured image data of the target area. The control device 110 reads the other vehicle icon image O4 indicating the back of the other vehicle from the ROM 12, and displays it superimposed on the captured image C4. Thus, the actual other vehicle V4 facing the front with respect to the rear camera 22 of the host vehicle V2 is displayed as the other vehicle icon image O4 facing the front with respect to the vehicle icon image V2 ^′ even in the virtual viewpoint video PV2 ^′. .

Note that the other vehicle image V3 ^′ is displayed with the back side facing the virtual viewpoint VP2 even in the virtual viewpoint video PV2 ^′ . That is, the actual other vehicle V3 facing the back with respect to the front camera 21 of the host vehicle V2 is displayed as the other vehicle image V3 ^′ facing the back with respect to the vehicle icon image V2 ^′ even in the virtual viewpoint video PV2 ^′ .

As described above, in the virtual viewpoint video PV2 ^′ displayed by the display device 100 according to the present embodiment, the other vehicle image V3 ^′ and the other vehicle icon image O4 viewed from the virtual viewpoint are viewed from the host vehicle V illustrated in FIG. It is displayed in the same direction as the actual other vehicles V3 and V4. Thereby, the passenger of the own vehicle V2 can simultaneously confirm the situation before and after the own vehicle V2, which is difficult to confirm from the own vehicle V2, through the virtual viewpoint video PV2 ^′ , and the direction of the object viewed from the own vehicle V2 can be accurately determined. Can grasp.

  As described above, the control device 110 of the present embodiment displays the direction of the other vehicle with respect to the camera 20 in the opposite direction using the other vehicle icon image indicating the other vehicle. Each other vehicle icon image showing the front and back of the other vehicle shows the actual feature points of the front and back of the other vehicle. Therefore, the direction of the other vehicle is indicated to the passenger of the own vehicle V through the virtual viewpoint image. Can be conveyed accurately.

  Moreover, the control apparatus 110 of this embodiment displays the direction of the passenger | crew's face and body of a two-wheeled vehicle with respect to the camera 20 in the opposite direction using the two-wheeled vehicle icon image which shows a two-wheeled vehicle. Since the two-wheeled vehicle icon images showing the front and back of the two-wheeled vehicle show the characteristic points of the front and rear of the actual two-wheeled vehicle occupant, the direction of the two-wheeled vehicle is accurately confirmed to the occupant of the own vehicle V through the virtual viewpoint image. Can tell.

  Furthermore, the control device 110 of the present embodiment changes the display of the object according to the direction of the object. When the target object is another vehicle, the control device 110 displays the direction of the other vehicle in the opposite direction. When the target object is a two-wheeled vehicle, the direction of the face and body of the occupant of the two-wheeled vehicle is not the two-wheeled vehicle itself. Is displayed in the opposite direction. Accordingly, the feature points of the target object can be appropriately displayed according to the attribute of the target object, and the direction of the target object according to the attribute of the target object can be accurately confirmed to the occupant of the host vehicle V through the virtual viewpoint video. Can tell.

«Third embodiment»
In the two embodiments described above, the case where the moving body is the host vehicle has been described as an example. However, in the third embodiment, the case where the moving body is a submarine explorer will be described as an example. Note that the moving body of the present embodiment may be a manned machine on which a human can be boarded, or an unmanned machine on which a human is not boarding.

  The display system of the present embodiment may be configured as a device mounted on a moving body, or may be configured as a portable device that can be brought into the moving body. Further, a part of the configuration of the display system according to the present embodiment may be mounted on a moving body, and another configuration may be mounted on a device physically different from the moving body, and the configuration may be distributed. In this case, the mobile body and another device are configured to be able to exchange information.

  Since the display device of the present embodiment has the same configuration as that of the above-described two embodiments, the description is incorporated.

FIG. 13 is a diagram illustrating an example of a virtual viewpoint video displayed by the display device according to the present embodiment. A virtual viewpoint video PV3 ^′ shown in FIG. 13 is an example of a virtual viewpoint video when the display device of the present embodiment is mounted on a seafloor explorer. For convenience of explanation, FIG. 13 shows a virtual viewpoint VP3. Note that the objects in the present embodiment include creatures such as fish and obstacles such as drifting objects. Moreover, in this embodiment, the icon image which shows living organisms, such as a fish, may be memorize | stored in ROM12 previously.

In the submarine spacecraft V6 (not shown), a number of cameras necessary for imaging the periphery of the seabed spacecraft are installed at predetermined positions. As illustrated in FIG. 13, the control device according to the present embodiment creates a virtual viewpoint video in which a cylindrical space model P3 is viewed from a predetermined virtual viewpoint. The space model P3 is composed of a submarine explorer icon image V6 ^′ located at the center of the cylindrical bottom surface, and a captured image (not shown) of a camera mapped on the cylindrical side surface.

The control device of the present embodiment specifies a region C5 (target region TA) on the projection plane in which the imaging direction of the camera of the seafloor explorer V6 and the direction of the virtual viewpoint face each other. The control device detects whether or not a target object (for example, a living organism such as a fish) exists in the target area TA, and when the target object exists, the direction of the target object with respect to the camera is reversed by using, for example, an icon image. The virtual viewpoint video PV3 ^′ displayed in FIG. Thereby, even when the display device is mounted on the submarine spacecraft, it is possible to accurately convey the direction of the object to the occupant of the seafloor spacecraft through the virtual viewpoint video PV3 ^′ .

<< Fourth Embodiment >>
In the fourth embodiment, a case where the moving body is a rotary wing machine will be described as an example. Note that the moving body of the present embodiment may be a manned machine on which a human can be boarded, or an unmanned machine on which a human is not boarding.

  The display system of the present embodiment may be configured as a device mounted on a moving body, or may be configured as a portable device that can be brought into the moving body. Further, a part of the configuration of the display system according to the present embodiment may be mounted on a moving body, and another configuration may be mounted on a device physically different from the moving body, and the configuration may be distributed. In this case, the mobile body and another device are configured to be able to exchange information.

  Since the display device of the present embodiment has the same configuration as that of the above-described three embodiments, the description is incorporated.

FIG. 14A is a diagram illustrating an example of a virtual viewpoint video displayed by the display device according to the present embodiment. A virtual viewpoint video PV4 ^′ shown in FIG. 14A is an example of a virtual viewpoint video in the case where the display device of the present embodiment is mounted on a manned aircraft on which the human body can ride, which is the rotary wing aircraft R1. Examples of the rotary wing aircraft R1 include a helicopter. FIG. 14B is a diagram illustrating another example of the virtual viewpoint video displayed by the display device according to the present embodiment. A virtual viewpoint video PV5 ^′ shown in FIG. 14B is an example of a virtual viewpoint video when the display device of the present embodiment is mounted on an unmanned aircraft that is a rotorcraft R2 and is not boarded by humans. An example of the rotary wing machine R2 is a drone.

  Note that the objects in the present embodiment include other rotorcraft, other aircraft, creatures such as birds, and the like. In the present embodiment, the ROM 12 may store in advance an icon image indicating a creature such as a rotary wing aircraft, an aircraft, or a bird. In the rotary wing aircraft, a number of cameras necessary for imaging the surroundings of the rotary wing aircraft are installed at predetermined positions.

As shown in FIG. 14A, the control device identifies a projection plane area C6 (target area TA) in which the imaging direction of the camera of the rotary wing machine R1 (rotary wing machine icon image R1 ^′ ) and the orientation of the virtual viewpoint VP4 face each other. To do. The control device detects whether or not a target object (for example, another aircraft) exists in the target area TA, and when the target object exists, the direction of the target object with respect to the camera is reversed by using, for example, an icon image. The displayed virtual viewpoint video PV4 ^′ is displayed on the display.

Further, as shown in FIG. 14B, the control device performs a projection plane area C7 (target area TA) in which the imaging direction of the camera of the rotary wing machine R2 (rotary wing machine icon image R2 ^′ ) and the orientation of the virtual viewpoint VP5 face each other. Is identified. The control device detects whether or not a target object (for example, another aircraft) exists in the target area TA, and when the target object exists, the direction of the target object with respect to the camera is reversed by using, for example, an icon image. The displayed virtual viewpoint video PV5 ^′ is displayed on the display.

As described above, even when the display device of the present invention is mounted on a rotary wing aircraft, the orientation of the object is indicated to the operator of the rotary wing aircraft or the crew of the rotary wing aircraft through the virtual viewpoint images PV4 ^′ and PV5 ^′. Can communicate accurately.

  The embodiment described above is described for facilitating the understanding of the present invention, and is not described for limiting the present invention. Therefore, each element disclosed in the above embodiment is intended to include all design changes and equivalents belonging to the technical scope of the present invention.

  For example, in the above-described embodiment, the configuration of changing the display of the target object by specifying the direction of the target object with respect to the camera 20 and displaying the icon image in the opposite direction to the specified target object direction is exemplified. However, the present invention is not limited to this. An icon image having the same direction as the direction of the specified object may be processed to create an icon image opposite to the direction of the specified object.

  For example, when the direction of the pedestrian with respect to the camera 20 is specified as the front, the front of the pedestrian icon image may be read from the ROM 12. Then, the face and body of the read pedestrian icon image may be painted with a predetermined color, and processing for superimposing the hair and the buttocks on the pedestrian icon image may be executed. Thereby, the pedestrian icon image which shows the back of a pedestrian can be displayed. For example, when the direction of the pedestrian with respect to the camera 20 is specified as the back side, the back side of the pedestrian icon image may be read from the ROM 12. And you may perform the process which superimposes eyes and a mouth on the back head of the read pedestrian icon image. Thereby, the pedestrian icon image which shows the front of a pedestrian can be displayed.

  Moreover, although embodiment mentioned above illustrated the structure which displays the direction of the target object with respect to the camera 20 in the opposite direction using an icon image, it is not limited to this. Image processing may be executed from the data of the object on the captured image, and the direction of the object with respect to the camera 20 may be displayed in the opposite direction.

  For example, as shown in FIG. 15A, the face of the pedestrian is detected from the captured image data, the direction of the pedestrian with respect to the camera 20 is specified as the front, and the color of the pedestrian's hair is also detected. Also good. Then, as shown in FIG. 15B, the pedestrian's face on the captured image may be painted with the detected pedestrian's hair color and displayed as an image showing the pedestrian's back. Since the face indicating the front of the pedestrian is not displayed, the occupant of the host vehicle V can recognize that the pedestrian is approaching the host vehicle V with the face facing the host vehicle V.

  Further, as shown in FIG. 15A, the upper body of the pedestrian is examined from the captured image data, the orientation of the pedestrian with respect to the camera 20 is specified as the front, and the color of the pedestrian's clothes is also detected. Also good. Then, as shown in FIG. 15B, the upper body of the pedestrian on the captured image may be painted with the color of the detected pedestrian clothes and displayed as an image showing the back of the pedestrian. Since buttons of clothes indicating the front of the pedestrian are not displayed, the passenger of the own vehicle V can recognize that the pedestrian is approaching the own vehicle V with the front facing the own vehicle V.

  FIG. 15 is a diagram illustrating an example of a display processing process for processing a pedestrian display. FIG. 15A shows an example of an image before processing the pedestrian display, and FIG. 15B shows an example of an image after processing the pedestrian display.

  Furthermore, in embodiment mentioned above, when the direction of the target object with respect to the camera 20 was specified with the back, the structure which changes and displays the direction of a target object to the front was illustrated, but it is not limited to this. The object whose orientation with respect to the camera 20 is identified as the back surface is actually facing away from the host vehicle V. Therefore, it may be determined that the risk that this object gives to the occupant of the host vehicle V is low. When the direction of the object with respect to the camera 20 is specified as the back, the direction of the object with respect to the camera 20 is specified as the front. Compared with the case where it does, you may lower the priority of the superimposition display by an icon image. Thereby, the passenger | crew of the own vehicle V can recognize the target object which may approach the own vehicle V through a virtual viewpoint image | video, and can reduce the load of image processing.

  In this specification, the display system 1 including the display device 100 as one embodiment of the display device according to the present invention will be described as an example. However, the present invention is not limited to this.

  In this specification, the control device 10 and the control device 110 including the CPU 11, the ROM 12, and the RAM 13 will be described as an embodiment of the controller according to the present invention.

  Further, in the present specification, as one aspect of the first icon of the present invention, as a pedestrian icon, as one aspect of the second icon of the present invention, as a motorcycle icon, as one aspect of the third icon of the present invention, Other vehicle icons will be described, but the present invention is not limited to this.

DESCRIPTION OF SYMBOLS 1 ... Display system 100 ... Display apparatus 10 ... Control apparatus 11 ... CPU
12 ... ROM
13 ... RAM
DESCRIPTION OF SYMBOLS 200 ... Mobile device 20 ... Camera 30 ... Controller 40 ... Crew operating device 50 ... Display

Claims (11)

Using a controller with a display control function,
Acquire the captured image captured by the camera mounted on the moving body,
The captured image is projected onto a three-dimensional shape that surrounds the periphery of the moving body, and the moving body and an image of the surroundings of the moving body are generated from a virtual viewpoint located outside the three-dimensional shape,
Specify the area of the projection plane where the imaging direction of the camera and the virtual viewpoint face each other as a target area,
Detecting whether an object is present in the target area;
If the object is present, identify the orientation of the object relative to the camera;
A control method for a display device, which displays the video in which the direction of the object is displayed in the opposite direction on a display. A control method for a display device according to claim 1,
When the direction of the object is the front, the back of the object is displayed,
The control method of the display apparatus which displays the front of the said target object when the direction of the said target object is a back surface. A control method for a display device according to claim 1 or 2,
The object is a pedestrian,
A control method for a display device, wherein a first icon indicating the pedestrian is used to display the face and body directions of the pedestrian relative to the camera in opposite directions. A control method for a display device according to claim 3,
A control method for a display device, wherein when the back of the pedestrian is detected, the back of the pedestrian is displayed with the first icon indicating a face. A control method for a display device according to claim 3,
A control method for a display device, wherein when the back of the upper body of the pedestrian is detected, the back of the upper body of the pedestrian is superimposed and displayed with the first icon indicating the front of the clothes. A control method for a display device according to claim 1 or 2,
The object is a pedestrian,
A control method for a display device, in which, when the face and hair of the pedestrian are detected, the face of the pedestrian is painted with the color of the pedestrian's hair. A control method for a display device according to claim 1 or 2,
The object is a pedestrian,
A control method for a display device, in which, when clothing of the upper body of the pedestrian is detected, the upper body of the pedestrian is painted with the color of the clothing of the pedestrian. A control method for a display device according to claim 1 or 2,
The object is a motorcycle and an occupant of the motorcycle,
A control method for a display device, wherein a second icon indicating the occupant of the motorcycle is used to display the face and body direction of the occupant of the motorcycle with respect to the camera in opposite directions. A control method for a display device according to claim 1 or 2,
The object is a vehicle;
A control method for a display device, wherein a third icon indicating the vehicle is used to display the direction of the vehicle in the opposite direction with respect to the camera. A control method for a display device according to any one of claims 1 to 9,
Detecting attributes of the object;
A control method for a display device, wherein display of the object is changed according to an attribute of the object. Display,
A display device comprising a controller,
The controller is
Acquire the captured image captured by the camera mounted on the moving body,
The captured image is projected onto a three-dimensional shape that surrounds the periphery of the moving body, and the moving body and an image of the surroundings of the moving body are generated from a virtual viewpoint located outside the three-dimensional shape,
Specify the area of the projection plane where the imaging direction of the camera and the virtual viewpoint face each other as a target area,
Detecting whether an object is present in the target area;
If the object is present, identify the orientation of the object relative to the camera;
A display device that displays the video in which the direction of the object is displayed in the opposite direction on the display.