JP6956376B2 - Display control system, display system, display control method, program, and mobile - Google Patents

Description

The present disclosure relates to a display control system, a display system, a display control method, a program, and a moving body, and more particularly to a display control system, a display system, a display control method, a program, and a moving body that display information presented to an occupant.

Conventionally, there is an information display device that captures the front of a vehicle with a camera, extracts a facility image from the actual image taken by the camera, adds facility information corresponding to the extracted facility image to the actual image, and displays it on the display device. (See, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2014-153095

The information display device having the above configuration automatically extracts one or a plurality of facility images shown in the actual image and displays them overlaid on the actual image. Therefore, there is also a problem that the facility information (presentation information) of the desired facility (object) is not displayed on the display device when the vehicle moves and the desired facility is out of the field of view of the camera.

An object of the present disclosure is to provide a display control system, a display system, a display control method, a program, and a moving body capable of presenting presentation information about an object intended by a user.

The display control system of one aspect of the present disclosure includes a first acquisition unit, a control unit, a second acquisition unit, and a third acquisition unit. The first acquisition unit acquires image data of an imaging device that photographs the surroundings of a moving body. The control unit causes the first display device to display an image around the moving body based on the image data acquired by the first acquisition unit. The second acquisition unit acquires designated information for designating an object to be displayed among the objects shown in the image displayed on the first display device. The third acquisition unit acquires the presentation information regarding the object to be displayed based on the designated information. The control unit displays the image of the object specified by the designated information acquired by the second acquisition unit on the second display device, and then the presentation acquired by the third acquisition unit. and displays the information on the second display device.

The display system of one aspect of the present disclosure includes the display control system and the first display device and the second display device whose display is controlled by the control unit.

The display control method of one aspect of the present disclosure includes a process of acquiring image data of an imaging device that captures the surroundings of a moving body, and displays an image of the surroundings of the moving body on a first display device based on the image data. The process, the process of acquiring the designated information for designating the object to be displayed among the objects shown in the image displayed on the first display device, and the process of acquiring the designated information for designating the object to be displayed, and the process of acquiring the designated information based on the designated information. a process of acquiring presentation information about the object, the image in which the target object specified is captured by the designation information after displaying on the second display device to display the presentation information to the second display device Processing and including.

The program of one aspect of the present disclosure is a program for causing a computer system to execute the display control method.

The mobile body of one aspect of the present disclosure includes the display system and a mobile body main body on which the display system is mounted.

According to the present disclosure, presentation information regarding an object intended by a user can be presented.

FIG. 1 is a block diagram of an entire system including a display control system according to an embodiment of the present disclosure. FIG. 2 is a schematic view of the vehicle interior of a mobile body including the display system according to the embodiment of the present disclosure. FIG. 3 is a flowchart illustrating the operation of the display control system described above. FIG. 4 is a schematic view showing an example of an image to be displayed on the first display device and the second display device of the same display system. FIG. 5 is a schematic view showing the operation of the same display system in the first display device. FIG. 6 is a schematic view showing a screen displayed on the second display device of the same display system. FIG. 7 is a schematic view showing presentation information displayed on the second display device of the same display system.

(Embodiment)
Hereinafter, the display control system 1, the display system 10, and the moving body (here, the automobile 100) according to the embodiment of the present disclosure will be described with reference to FIG. The display control system 1 of the present embodiment includes a first acquisition unit 11, a second acquisition unit 12, a third acquisition unit 13, and a control unit 14. Further, the display system 10 of the present embodiment includes a display control system 1, a first display device 31, and a second display device 32. Further, the automobile 100 of the present embodiment includes a display system 10 and a moving body main body 102 (see FIG. 2).

As shown in FIG. 1, the display system 10 according to the present embodiment includes the first acquisition unit 11, the second acquisition unit 12, the third acquisition unit 13, the first display device 31 and the second display device 32. A control unit 14 for controlling the display is provided.

The display control system 1 of the present embodiment is applied to, for example, an automobile 100 (see FIG. 2) equipped with an automatic driving system capable of automatically driving without a human performing a driving operation. The display system 10 presents information to the user (occupant) of the automobile 100.

The automatic driving system of the automobile 100 includes a driving control unit 2 and a detection system 4.

The detection system 4 includes, for example, an image pickup device 41 and a laser radar 42.

The image pickup device 41 is an omnidirectional camera including image pickup elements such as a CCD (Charge-Coupled Device) image sensor and a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and photographs the periphery of the automobile 100 over the entire circumference.

The laser radar 42 irradiates the space around the automobile 100 with a pulsed laser beam, and receives the reflected light reflected by an object existing around the automobile 100. The laser radar 42 detects the distance to the object based on the time from emitting the laser beam to receiving the reflected light.

Further, the detection system 4 obtains the current position of the automobile 100 by using, for example, GPS (Global Positioning System), and detects the position information regarding the position of the automobile 100 based on the current position of the automobile 100. The detection system 4 acquires map information around the current position based on the current position of the automobile 100. The detection system 4 may acquire map information around the current position from a memory that stores map information, or a server when the detection system 4 or the mobile communication unit included in the automobile 100 communicates with an external server 200. Map information may be obtained from 200. Here, the position information is, for example, information on a road (traffic route) at the current position of the automobile 100. The position information includes, for example, information such as the number of lanes on the road, the width of the road, the presence or absence of a sidewalk, the slope, the curvature of a curve, the presence or absence of a sidewalk, and whether or not the current position is an intersection (an intersection such as a crossroad or a junction). Information, information on whether the road is one-way or not, etc.

Further, the detection system 4 may acquire vehicle information regarding the state of the automobile 100. The vehicle information is information representing the local state of the automobile 100 itself, and is information that can be detected by a sensor mounted on the automobile 100. Specific examples of vehicle information include the moving speed (running speed) of the automobile 100, the acceleration applied to the automobile 100, the steering angle, and the pulse, facial expression, and line of sight of the driver (occupant) detected by the driver monitor. Further, data unique to the automobile 100 such as the vehicle width, the vehicle height, and the total length are also included in the vehicle information.

The driving control unit 2 determines a traveling route based on the current position information acquired by the detection system 4 and the destination information, and controls the automobile 100 so as to automatically travel according to the traveling route.

In the present embodiment, the first display device 31 is composed of three display devices 311, 312, 313 (see FIG. 2). The first display device 31 (display device 311, 312, 313) and the second display device 32 are composed of a thin display such as a liquid crystal display or an organic EL (Electro Luminescence) display, for example.

The display devices 311, 312, 313 and the second display device 32 are provided at positions that are easily visible to the user inside the automobile 100. In this embodiment, the display devices 311, 312, 313 and the second display device 32 are arranged on the dashboard 101. Further, in the present embodiment, the display devices 311, 312, 313 are arranged side by side in the left-right direction on the dashboard 101. The display device 311 is arranged at the center in the left-right direction, the display device 312 is arranged on the left side of the display device 311 and the display device 313 is arranged on the right side of the display device 311. Further, the second display device 32 is arranged below the display device 311 arranged at the center.

The first acquisition unit 11 acquires the image data of the image pickup device 41 that photographs the surroundings of the automobile 100.

The second acquisition unit 12 accepts the operation of the user (occupant). The second acquisition unit 12 includes, for example, a capacitance type or piezoelectric type touch sensor provided on the screens of the first display device 31 (display devices 31 to 313) and the second display device 32, respectively. The second acquisition unit 12 receives an operation (for example, an operation such as tapping, flicking, swiping, etc.) in which the user touches the screens of the first display device 31 (display devices 311, 312, 313) and the second display device 32. In the present embodiment, the second acquisition unit 12 specifies that the user specifies the object to be displayed among the objects shown in the image displayed on the first display device 31 (display devices 31 to 313). Get information.

The third acquisition unit 13 acquires the presentation information regarding the object to be displayed based on the designated information acquired by the second acquisition unit 12, for example, by communicating with the server 200.

The control unit 14 is composed of, for example, a CPU (Central Processing Unit) and a microcomputer having a memory as a main configuration. In other words, the control unit 14 is realized by a computer having a CPU and a memory, and the computer functions as the control unit 14 by executing a program stored in the memory by the CPU. Although the program is recorded in advance in the memory of the control unit 14, the program may be provided by being recorded in a telecommunication line such as the Internet or recorded in a recording medium such as a memory card. The same applies to the operation control unit 2.

The control unit 14 controls the display of the first display device 31 (display devices 311, 312, 313) and the second display device 32. Further, the control unit 14 causes the second display device 32 to display the presentation information acquired by the third acquisition unit 13.

Next, the operation of the display system 10 will be described with reference to the flowchart of FIG.

The display system 10 repeatedly executes a process of displaying an image on the first display device 31 and the second display device 32 at a predetermined display interval (for example, a time interval of 1/60 second).

When the first acquisition unit 11 acquires image data from the image pickup device 41 (S1), the control unit 14 displays an image of the periphery of the automobile 100 on the first display device 31 (display devices 31 to 313) (S2). The central display device 311 of the display devices 311 to 313 displays an image of the space in front of the automobile 100 among the images taken by the image pickup device 41. Further, the display device 312 displays an image of the space on the left side of the automobile 100 among the images taken by the image pickup device 41, and the display device 313 displays the image of the automobile 100 among the images taken by the image pickup device 41. The image of the space on the right is displayed. Therefore, the three display devices 31 to 313 display an image in a range of 180 degrees forward when viewed from the center of the automobile 100. When the user touches the screen of the display device 311 and swipes to the right, the control unit 14 switches to the three display devices 311 to 313 based on the operation information acquired by the second acquisition unit 12. Move the range of the displayed image to the left. Further, when the user touches the screen of the display device 311 and swipes to the left, the control unit 14 displays the three display devices 31 to 13 based on the operation information acquired by the second acquisition unit 12. Move the range of the displayed image to the right.

Further, the control unit 14 causes the second display device 32 to display an image image P2 representing a range of images displayed on the display devices 31 to 13 (see FIG. 4). Specifically, the first icon A1 and the circular second icon A2 surrounding the first icon A1 are displayed on the second display device 32. The first icon A1 is a figure representing a state in which the automobile 100 is viewed from above. The second icon A2 represents a range of images displayed on the display devices 311 to 313. Here, the upper semicircular ring region of the second icon A2 is highlighted.

The control unit 14 determines whether or not the second acquisition unit 12 has acquired the designated operation (S3). Here, the designation operation is an operation performed by the user to specify the object to be displayed among the objects shown in the image P1 displayed on the first display device 31 (display devices 31 to 313). Is. For example, as shown in FIG. 5, the second display device 32 is located in the direction in which the second display device 32 is located, that is, downward (direction of arrow D1 in FIG. 5) by touching the position of the object B1 shown in the screen of the display device 311 with the finger 300. When an operation (swipe operation) is performed, the second acquisition unit 12 displays information indicating the position where the swipe operation is performed, and designated information for designating the object B1 (for example, a building such as a commercial building) to be displayed. Is output to the control unit 14. The designated operation is not limited to the swipe operation, and may be an operation such as tapping, long tapping, or flicking. Further, in the present embodiment, the first display device 31 is composed of three display devices 31 to 313, and the designated operation may be performed by any of the three display devices 31 to 313. Further, the object specified by the user is not limited to a commercial building, and may be a commercial facility such as a supermarket, a public facility such as a station or a stadium, or a bronze statue.

When the second acquisition unit 12 receives the designated operation (S3: Yes), the control unit 14 causes the second display device 32 to display the image P21 that captures the image P1 of the display device 311. Further, the control unit 14 transmits the image of the object B1 corresponding to the operation position and the position information of the current position of the automobile 100 to the server 200 based on the designated information. When the server 200 receives the image and the position information of the object B1 from the automobile 100, the server 200 performs an image search, identifies the object B1, creates presentation information about the object B1, and transfers the created presentation information to the automobile 100. Send. Here, the presented information is information about the object B1, and when the object B1 is a commercial building, information such as the name, address, telephone number, and business hours of the commercial building is included.

When the third acquisition unit 13 acquires the presentation information transmitted from the server 200 (S4), the control unit 14 causes the second display device 32 to display the image P22 indicating the presentation information (S5). Here, the image P22 showing the presentation information is composed of an image P221 showing the whole view of the object B1 and an image P222 displaying information such as the name, address, telephone number, and business hours of the object B1. As a result, the user can see the information about the object B1. Further, even if the automobile 100 moves and the object to be displayed is removed from the screen of the first display device 31, the presentation information is displayed on the second display device 32, so that the object intended by the user is presented. The information can be displayed on the second display device 32, and the display can be easily seen by the user. Further, as compared with the case where the presentation information is automatically presented to all the buildings and the like in the screen of the first display device 31, the presentation information of only the object specified by the user is displayed on the second display device 32. Therefore, there is an advantage that the display of the presented information becomes easy to see.

The server 200 searched based on the image and the position information transmitted from the automobile 100, but when there was no presentation information to be presented to the user, the server 200 could not acquire the presentation information for the automobile 100. All you have to do is send the information indicating that. When the third acquisition unit 13 of the display control system 100 cannot acquire the presentation information from the server 200, the control unit 14 displays on the second display device 32 that there is no presentation information, or a designated place. It suffices to display the map information etc. around the.

If the designated operation is not acquired in step S3, the control unit 14 ends the process, and when the predetermined display interval elapses, the process of step S1 is started again.

Here, when the presentation information searched by the server 200 is information on an object different from the object B1 specified by the user, the user displays the presentation information on the screen of the second display device 32 in order to change the presentation information. Perform touch operations (tap, flick, swipe, etc.). When the second acquisition unit 12 acquires the operation information of the operation of touching the screen of the second display device 32, the control unit 14 performs a process of changing the presented information based on the operation information acquired by the second acquisition unit 12. .. The operation for changing the presented information is not limited to the operation of touching the screen of the second display device 32, and may be an operation using a switch provided separately. For example, the control unit 14 causes the second display device 32 to display a map around the current position, and when the user performs an operation of designating the display target in the map, the control unit 14 designates the display target on the server 200. Send information to be done. When the server 200 receives the information for designating the display target from the automobile 100, the server 200 creates presentation information regarding the display target based on the received information, and transmits the created presentation information to the automobile 100.

When the third acquisition unit 13 acquires the presentation information transmitted from the server 200, the control unit 14 causes the second display device 32 to display an image showing the presentation information. As a result, the control unit 14 can change the presentation information to be displayed on the second display device 32, and the user can see the information about the desired object.

(3) Modified Example The above embodiment is only one of various embodiments of the present disclosure. The above-described embodiment can be changed in various ways depending on the design and the like as long as the object of the present disclosure can be achieved. Further, the same functions as those of the display control system 1 and the display system 10 may be realized by a display control method, a program, a non-temporary recording medium on which the program is recorded, or the like. The display control method according to one aspect includes the following first to fifth processes. The first process is a process of acquiring image data of the image pickup apparatus 41 that photographs the surroundings of the moving body 100. The second process is a process of displaying an image around the moving body 100 on the first display device 31 based on the image data. The third process is a process of acquiring designated information for designating the object to be displayed among the objects shown in the image displayed on the first display device 31. The fourth process is a process of acquiring presentation information about the object to be displayed based on the designated information. The fifth process is a process of displaying the presented information on the second display device 32. The (computer) program according to one aspect is a program for causing a computer system to execute a display control method.

Hereinafter, modifications of the above embodiment will be listed. The modifications described below can be applied in combination as appropriate.

The execution body of the display control system 1, the display system 10, or the display control method in the present disclosure includes a computer system. The main configuration of a computer system is a processor and memory as hardware. When the processor executes the program recorded in the memory of the computer system, the function as the execution subject of the display control system 1, the display system 10, or the display control method in the present disclosure is realized. The program may be pre-recorded in the memory of the computer system, may be provided through a telecommunication line, or may be recorded and provided on a non-temporary recording medium readable by the computer system. Non-temporary recording media of this type include memory cards, optical disks, hard disk drives, and the like. A processor in a computer system is composed of one or more electronic circuits including a semiconductor integrated circuit (IC) or a large scale integrated circuit (LSI). A plurality of electronic circuits may be integrated on one chip, or may be distributed on a plurality of chips. The plurality of chips may be integrated in one device, or may be distributed in a plurality of devices.

Further, the functions of the first acquisition unit 11, the second acquisition unit 12, the third acquisition unit 13, and the control unit 14 of the display control system 1 may be distributed to two or more systems. Further, the function of the control unit 14 of the display control system 1 may be realized by, for example, the cloud (cloud computing).

Further, the first display device 31 is composed of three display devices 311, 312, 313, but the first display device 31 is composed of one or a plurality of display devices capable of displaying an image in a desired range. You may be.

Further, the display system 10 may include a plurality of second display devices 32 for displaying the presented information. In this case, if you put your finger on the position of the object shown in the screen of the first display device 31 (display devices 311, 312, 313) and swipe in the direction of the second display device where you want to display the presented information. , The control unit 14 causes the second display device in the direction in which the swipe operation is performed to display the presented information. Further, the display system 10 may display the presentation information on a display device other than the second display device 32, and may display the presentation information on the display device in the direction instructed by the user's operation.

Although the third acquisition unit 13 of the display control system 1 acquires the presentation information from the server 200 outside the automobile 100, the presentation information may be acquired from other than the server 200. For example, the third acquisition unit 13 includes vehicle-to-vehicle communication (V2V: Vehicle to Vehicle) between the automobile 100 and peripheral vehicles, and vehicle-to-vehicle / road-to-vehicle communication (V2X: Vehicle to) between the automobile 100 and peripheral vehicles or infrastructure. The presentation information may be obtained by (Everything) or the like. Further, the presentation information is stored in the storage unit of the display control system 1 or the automobile 100, and the third acquisition unit 13 may acquire the presentation information from the storage unit of the display control system 1 or the automobile 100.

Further, the display control system 1 and the display system 10 are not limited to those used for the automobile 100, and can be applied to moving objects other than the automobile 100, such as two-wheeled vehicles, trains, aircrafts, construction machines, and ships.

(summary)
As described above, the display control system (1) of the first aspect includes the first acquisition unit (11), the second acquisition unit (12), the third acquisition unit (13), and the control unit (14). ) And. The first acquisition unit (11) acquires the image data of the image pickup apparatus (41) that captures the surroundings of the moving body (100). The control unit (14) causes the first display device (31,311 to 313) to display an image around the moving body (100) based on the image data acquired by the first acquisition unit (11). The second acquisition unit (12) acquires designated information for designating the object (B1) to be displayed among the objects shown in the image displayed on the first display device (31,311 to 313). do. The third acquisition unit (13) acquires the presentation information regarding the object (B1) to be displayed based on the designated information. The control unit (14) causes the second display device (32) to display the presentation information acquired by the third acquisition unit (13).

As a result, the presentation information about the object (B1) to be displayed corresponding to the designated information is displayed on the second display device (32), so that the presentation information about the object (B1) intended by the user can be presented. can.

Further, since the presentation information is displayed on the second display device (32) different from the first display device (31,311 to 313) that displays the surrounding image, the moving body (100) moves and the first display device (100) moves. Even if the object (B1) to be displayed is removed from the screen of the display device (31, 313-1-13), the presented information is displayed on the second display device (32), so that the object intended by the user (the object (B1) intended by the user) is displayed. The presented information of B1) can be displayed on the second display device (32).

The display system of the second aspect includes the display control system (1) of the first aspect, the first display device (31,311 to 313) and the second display device (31,311 to 313) whose display is controlled by the control unit (14). 32) and.

As a result, the presentation information about the object (B1) to be displayed corresponding to the designated information is displayed on the second display device (32), so that the presentation information about the object (B1) intended by the user can be presented. can.

The display control method of the third aspect includes a process of acquiring image data of the image pickup apparatus (41) that captures the surroundings of the moving body (100), and a process of acquiring an image of the surroundings of the moving body (100) based on the image data. 1 Of the processing to be displayed on the display device (31,311 to 313) and the object (B1) shown in the image displayed on the first display device (31,311 to 13), the object to be displayed. A process of acquiring designated information for designating (B1), a process of acquiring presentation information about an object (B1) to be displayed based on the designated information, and a process of displaying the presented information on the second display device (32). And, including.

As a result, the presentation information about the object (B1) to be displayed corresponding to the designated information is displayed on the second display device (32), so that the presentation information about the object (B1) intended by the user can be presented. can.

The display control method of the fourth aspect is a program for causing a computer system to execute the display control method of the fourth aspect.

The mobile body (100) of the fifth aspect includes the display system (10) of the second aspect and the mobile body main body (102) on which the display control system (1) is mounted.

As a result, the presentation information about the object (B1) to be displayed corresponding to the designated information is displayed on the second display device (32), so that the presentation information about the object (B1) intended by the user can be presented. can.

1 Display control system 10 Display system 11 1st acquisition unit 12 2nd acquisition unit 13 3rd acquisition unit 14 Control unit 31 1st display device 32 2nd display device 41 Imaging device 100 Automobile (mobile)
102 Mobile body 200 Server 311-13 Display device (1st display device)
B1 object

Claims (5)

The first acquisition unit that acquires the image data of the image pickup device that captures the surroundings of the moving object, and
A control unit that causes the first display device to display an image around the moving body based on the image data acquired by the first acquisition unit.
Among the objects shown in the image displayed on the first display device, the second acquisition unit for acquiring the designated information for designating the object to be displayed, and the second acquisition unit.
It is provided with a third acquisition unit that acquires presentation information about the object to be displayed based on the designated information.
The control unit displays the image of the object specified by the designated information acquired by the second acquisition unit on the second display device, and then the presentation acquired by the third acquisition unit. display control system for displaying information on the second display device. A display system including the display control system according to claim 1, and the first display device and the second display device whose display is controlled by the control unit. The process of acquiring image data of an imaging device that captures the surroundings of a moving object, and
A process of displaying an image around the moving body on the first display device based on the image data, and
Among the objects displayed in the image displayed on the first display device, a process of acquiring designated information for designating the object to be displayed, and
A process of acquiring presentation information about the object to be displayed based on the designated information, and
The image in which the object designated by the designation information is captured after displaying on the second display device, the display control method comprising a process of displaying the presentation information to the second display device. For computer systems
A program for executing the display control method according to claim 3. The display system according to claim 2 and
A mobile body including a mobile body main body equipped with the display control system.

Images