EP3693930A1 - Image generation/recording/display device for moving body, and program - Google Patents
Image generation/recording/display device for moving body, and program Download PDFInfo
- Publication number
- EP3693930A1 EP3693930A1 EP18864949.5A EP18864949A EP3693930A1 EP 3693930 A1 EP3693930 A1 EP 3693930A1 EP 18864949 A EP18864949 A EP 18864949A EP 3693930 A1 EP3693930 A1 EP 3693930A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- moving body
- virtual
- dimensional video
- host
- virtual space
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000010276 construction Methods 0.000 claims abstract description 16
- 230000000694 effects Effects 0.000 claims abstract description 12
- 238000000034 method Methods 0.000 abstract description 19
- 230000002093 peripheral effect Effects 0.000 description 65
- 238000012545 processing Methods 0.000 description 44
- 238000009877 rendering Methods 0.000 description 15
- 238000005259 measurement Methods 0.000 description 11
- 238000003909 pattern recognition Methods 0.000 description 6
- 238000006243 chemical reaction Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000003384 imaging method Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 2
- 230000002194 synthesizing effect Effects 0.000 description 2
- 240000004050 Pentaglottis sempervirens Species 0.000 description 1
- 235000004522 Pentaglottis sempervirens Nutrition 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/10—Geometric effects
- G06T15/20—Perspective computation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/003—Navigation within 3D models or images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/50—Lighting effects
- G06T15/503—Blending, e.g. for anti-aliasing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T13/00—Animation
- G06T13/20—3D [Three Dimensional] animation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/08—Volume rendering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V20/584—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads of vehicle lights or traffic lights
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/588—Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
-
- G—PHYSICS
- G07—CHECKING-DEVICES
- G07C—TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
- G07C5/00—Registering or indicating the working of vehicles
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/24—Indexing scheme for image data processing or generation, in general involving graphical user interfaces [GUIs]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2210/00—Indexing scheme for image generation or computer graphics
- G06T2210/61—Scene description
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2210/00—Indexing scheme for image generation or computer graphics
- G06T2210/62—Semi-transparency
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2215/00—Indexing scheme for image rendering
- G06T2215/16—Using real world measurements to influence rendering
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/16—Anti-collision systems
- G08G1/166—Anti-collision systems for active traffic, e.g. moving vehicles, pedestrians, bikes
Definitions
- the present invention relates to a device which generates an image of an object in a periphery of a moving body to record or display the image and to a program for performing such operations.
- the disclosed technique is made in view of the aforementioned problems of the conventional art and an object is to provide information in a form easier to understand for a driver when an environment around a moving body driven by the driver is displayed as a video and to record a travel state of the moving body with a smaller storage capacity.
- ghosts image lag
- a past video, or the like may not be displayed for the host moving body, an oncoming moving body, a moving body whose absolute value of speed relative to the host moving body or absolute value of speed change relative to the host moving body is smaller than a predetermined value, a moving body whose speed is lower than the host moving body, a stationary object, or the like.
- ghosts, a past video, or the like may not be displayed for some of these moving bodies and object.
- an effect of display may vary depending on the types or characteristics of the moving bodies and object.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Graphics (AREA)
- Software Systems (AREA)
- Multimedia (AREA)
- General Engineering & Computer Science (AREA)
- Computer Hardware Design (AREA)
- Signal Processing (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Computing Systems (AREA)
- Geometry (AREA)
- Processing Or Creating Images (AREA)
- Traffic Control Systems (AREA)
- Time Recorders, Dirve Recorders, Access Control (AREA)
- Image Generation (AREA)
- Closed-Circuit Television Systems (AREA)
Abstract
Description
- The present invention relates to a device which generates an image of an object in a periphery of a moving body to record or display the image and to a program for performing such operations.
- There is an image conversion technique in which images captured by using cameras installed in a moving body such as an automobile are processed and converted to images similar to images of the moving body as viewed from various directions. There is a technique such as so-called "around view monitor" (registered trademark) in which images captured by using multiple cameras mounted in a moving body are synthesized to generate a bird's-eye view image of a region of a certain range from the moving body. Such an image synthesizing technique is used, for example, in driving in reverse, doing parallel parking, and the like to display, on a monitor, other moving bodies, objects, and the like at positions hard to see from a driver seat of the moving body and present them to the driver.
- In the image synthesizing technique as described above, the actual captured image is converted and displayed on a display device in the vehicle. Accordingly, when a target of image capturing is temporarily blocked by another moving body or moves to a blind spot of the camera, this target cannot be displayed in some cases.
- Moreover, there is a drive recorder which records traveling of a moving body as a video. One drive recorder is generally installed near a windshield of the moving body and a large-capacity storage device is necessary to record the video.
- The conventional art is described below.
- There is a technique as follows (for example, see Patent Document 1). An apparatus includes an input device, a virtual model database, a virtual model conversion device, a virtual model reconfiguration device, a virtual model processing device, and a control device or a display device. The virtual model conversion device recognizes information on multiple targets obtained from the input device, identifies a virtual model corresponding to each target in the virtual model database, and substitutes the targets with the virtual models. The virtual model reconfiguration device reconfigures the targets and relationships between the targets to virtual models and relationships between the virtual models corresponding thereto in a virtual space. The virtual model processing device understands and determines the virtual models and the relationships between the virtual models based on the reconfigured virtual models and the relationships between the virtual models and gives instructions to the control device or the display device.
- Moreover, there is a technique as follows (for example, see Patent Document 2). A navigation device generates a traveling road shape by using map information stored in the navigation device. This can achieve more realistic road display. Moreover, the navigation device performs calculation such that a display position of a host vehicle is fixed to a predetermined one point in a display and performs operation such that a mark indicating the position of the host vehicle, a mark indicating the position of a following vehicle, and a mark indicating the position of a peripheral vehicle other than the following vehicle are displayed to be superimposed on a road. This allows a user to easily determine a distance interval between the host vehicle and the peripheral vehicle.
- Moreover, there is a technique as follows (for example, see Patent Document 3). When the positions of other vehicles relative to a host vehicle are to be displayed, an image can be displayed by using image information of CCD cameras and the like. In this case, the vehicle types and the like of the other vehicles are received from data in communication processing and three-dimensional information of the other vehicles and map information (terrain information) of the periphery are read from a database. Then, three-dimensional models of the vehicles and the terrain are arranged and rendered in real time to match measurement results, based on detection results of the positions of the other vehicles relative to the host vehicle. This allows the vehicles and the terrain to be displayed on a display device as an artificial reality virtual image. The viewpoint of the virtual image is movable and an image from a freely-moved viewpoint can be displayed. For example, it is possible to display an operation lever in a portion of a display screen and move the viewpoint by operating the operation lever in any of directions of up, down, left, and right. Moving the viewpoint upward allows a user to three-dimensionally grasp an overall condition from a high position and three-dimensionally check presence of a vehicle in front which is blocked from view by a vehicle just in front of the host vehicle. Moreover, when the viewpoint is moved to such a position that the vehicles and the like are viewed from the side of the road, states of the host vehicle and the other vehicles in front of and behind the host vehicle can be displayed from the side from the third person point of view.
- Moreover, there is a technique in which viewpoint converted image generation means synthesizes a vehicle model image, rendered to be modeled on a vehicle, with a portion of a viewpoint converted image corresponding to the vehicle (for example, see Patent Document 4).
-
- Patent Document 1: Japanese Patent Application Publication No.
2003-216981 - Patent Document 2: Japanese Patent Application Publication No.
Hei 11-250396 - Patent Document 3: Japanese Patent Application Publication No.
2006-107521 - Patent Document 4: Japanese Patent Application Publication No.
2011-182059 - In techniques in which videos of a periphery of a moving body captured by multiple cameras are synthesized as described above, a portion outside the fields of view of the cameras is sometimes not displayed. Moreover, there is a limit to displaying the video of the periphery of the moving body from a freely-moved viewpoint.
- In techniques of constructing a virtual space by using virtual models as described above, there are needs for displaying the virtual models as a two-dimensional image in a manner easily understandable for the operator (driver).
- Regarding a drive recorder mounted in a moving body, only one drive recorder is installed near the windshield in most cases. Thus, the drive recorder is used to record only a video in a forward direction of the moving body and does not record videos in the other directions. Moreover, a large capacity recording medium is necessary to always record the video during traveling. When multiple cameras are installed to record the situation of the periphery of the moving body, a storage device with an even larger capacity is necessary.
- The disclosed technique is made in view of the aforementioned problems of the conventional art and an object is to provide information in a form easier to understand for a driver when an environment around a moving body driven by the driver is displayed as a video and to record a travel state of the moving body with a smaller storage capacity.
- The disclosed technique provides a moving body position information recognition display device including: a traveling state obtaining part configured to obtain information on a traveling state of a host moving body including a position and speed of the host moving body; an object information obtaining part configured to obtain information on a plurality of objects including a shape of a travel road and another moving body present in a periphery of the host moving body; a virtual space construction part configured to construct a virtual space by associating at least the host moving body and the plurality of objects respectively with virtual models and arranging the virtual models in a three-dimensional space based on the information obtained from the traveling state obtaining part and the object information obtaining part; and a two-dimensional video generation part configured to generate a two-dimensional video capturing the virtual space from a virtual camera arranged in the virtual space and output the two-dimensional video to a display screen. The two-dimensional video generation part generates the two-dimensional video while adding a predetermined effect to display of the virtual model corresponding to each of the objects in the two-dimensional video depending on speed of the object relative to the host moving body.
- The disclosed technique can provide information in a form easier to understand when an environment around a host moving body is displayed as a video and record a travel state of the moving body with a smaller storage capacity.
-
- [
Figs. 1A and 1B] Figs. 1A and 1B is a view illustrating examples of a virtual space and sensors installed in an actual moving body in one embodiment. - [
Fig. 2] Fig. 2 is a functional block diagram of one embodiment. - [
Fig. 3] Fig. 3 is a view illustrating an example of a first display image in one embodiment. - [
Fig. 4] Fig. 4 is a view illustrating a modified example of the first display image in one embodiment. - [
Fig. 5] Fig. 5 is a view illustrating an example of a second display image in one embodiment. - [
Fig. 6] Fig. 6 is a first flowchart of displaying ghosts of a peripheral moving body in one embodiment. - [
Fig. 7] Fig. 7 is a second flowchart of displaying the ghosts of the peripheral moving body in one embodiment. - [
Fig. 8] Fig. 8 is a diagram illustrating a hardware configuration. -
Figs. 1A and 1B are a view illustrating examples of a virtual space and sensors installed in an actual moving body in one embodiment. - A
virtual space 150 illustrated inFig. 1A is constructed based on information such as video information on peripheral objects sent from cameras (101a to 101d) installed in an actual movingbody 100R illustrated inFig. 1B , information on relative speed, positions, and the like of the peripheral objects obtained by aradar 103, information of GPS received by aradio antenna 105, various kinds of information sent from other systems connected to a radio network, and information on the actual movingbody 100R obtained from sensors (not illustrated) such as a speedometer, an accelerometer, and a gyroscope installed in the movingbody 100R. - The cameras (101a to 101d), the
radar 103, theradio antenna 105, and the sensors (not illustrated) such as the speedometer described above are examples and other appropriate information gathering devices may be used. Moreover, these devices for gathering information may be installed inside the actual movingbody 100R or outside the actual movingbody 100R. - In the
virtual space 150 inFig. 1A , avirtual camera 130 is desirably set such that a virtual movingbody 100 corresponding to the actual movingbody 100R illustrated inFig. 1B is located at substantially the center in an imaging direction of thevirtual camera 130. In thevirtual camera 130, various parameters can be set, which are used by a virtual imaging system of thevirtual camera 130 for conversion of thevirtual space 150 to a two-dimensional image. The parameters of thevirtual camera 130 include, for example, the position, imaging direction, focal distance, angle of field, and the like of thevirtual camera 130 in the virtual space. The depth of field is a parameter indicating depth in which objects are in focus in a virtual lens system and, in the embodiment, is desirably set to pan focus (specifically, set such that virtual objects present within any distance from thevirtual camera 130 are in focus). - In
Fig. 1A , stationary objects, other moving bodies traveling in a periphery of the actual movingbody 100R in an actual space, and the like are recognized from the aforementioned various kinds of information. In the embodiment of the present invention, the other moving bodies, stationary objects, and the like in the actual space are respectively associated with virtual models with reference to a 3D model database and the virtual models are arranged in thevirtual space 150. - In the virtual space of
Fig. 1A , atruck 110, anautomobile 112, abus 114, and aroadside pole 120 are arranged in the periphery of the virtual movingbody 100 in the virtual space corresponding to the actual movingbody 100R. These virtual models present in thevirtual space 150 correspond to the respective objects present in the actual space with respect to the virtual movingbody 100 and constantly move relative to the virtual movingbody 100 in thevirtual space 150 depending on the movement of the actual objects relative to the actual movingbody 100R. - In the embodiment, it is desirable that the virtual imaging system of the
virtual camera 130 moves in the same direction as the virtual movingbody 100 such that the virtual movingbody 100 is always arranged at the center of the screen and the virtual imaging system follows the virtual movingbody 100 such that the virtual movingbody 100 is arranged at the center of the screen. Note that it is desirable that the relative distance between thevirtual camera 130 and the virtual movingbody 100 and the parameters of the imaging system of the virtual camera can be freely set based on instructions of an operator (driver) riding in the actual movingbody 100R. A display device is desirably installed near a driver seat such that the operator (driver) or the like of the actual movingbody 100R can view a video captured by thevirtual camera 130. - Note that, for example, recognized two-wheelers and humans can also be placed in the virtual space. Accordingly, in the embodiment, it is desirable that comprehensiveness of recognizable objects is guaranteed for actual objects which need to be recognized in driving of the driver so that such objects can be arranged in the virtual space.
- Although oncoming vehicles of the actual moving
body 100R are not illustrated in the virtual space ofFig. 1A , the oncoming vehicles are also desirably included in the recognition targets. -
Fig. 2 is a functional block diagram of one embodiment. A movingbody measurement part 210 receives information from the sensors (not illustrated) such as the speedometer, the accelerometer, and the gyroscope installed in the actual movingbody 100R. The information on the actual movingbody 100R obtained in the movingbody measurement part 210 is sent to a virtualspace construction part 250. - A peripheral
object measurement part 220 receives the video information on the other moving bodies and objects in the periphery sent from the cameras (101a to 101d) installed in the actual movingbody 100R, the information on relative speed, positions, and the like of the other moving bodies and objects in the periphery obtained by theradar 103, the information of GPS received by theradio antenna 105, and the various kinds of information (road information, videos from cameras installed on roads, and the like) from other systems connected to the radio network. The information received by the peripheralobject measurement part 220 is sent to the virtualspace construction part 250 and anobject recognition part 230. - The
object recognition part 230 recognizes the other moving bodies present in the periphery and the objects such as poles on the road based on the information received by the peripheralobject measurement part 220 and sends results of the recognition to the virtualspace construction part 250. - The virtual
space construction part 250 obtains the shape of a road from acar navigation database 270 and creates a model of the road in thevirtual space 150. Moreover, the virtualspace construction part 250 arranges the virtual movingbody 100 corresponding to the actual movingbody 100R on the aforementioned virtual road created in thevirtual space 150 based on the information from the movingbody measurement part 210 and causes the virtual movingbody 100 to travel on the virtual road at speed corresponding to the actual movingbody 100R. The three-dimensional shape of the virtual movingbody 100 may be stored in the virtualspace construction part 250 or obtained from a3D model database 240. - Furthermore, the virtual
space construction part 250 obtains the virtual models corresponding to the other moving bodies and objects in the periphery from the 3D model database based on the information obtained from theobject recognition part 230. The virtualspace construction part 250 arranges the obtained virtual models corresponding to the other moving bodies and objects in the periphery in the virtual space. The virtualspace construction part 250 causes the virtual models corresponding to the other moving bodies and objects in the periphery to travel on the virtual road based on the information of the peripheralobject measurement part 220. - Moreover, the virtual
space construction part 250 sends the information on the virtual space to arendering part 260 and arecorder part 280. - The
rendering part 260 sets the parameters of thevirtual camera 130 based on virtual camera instruction information given by the operator (driver) and sends the information on imaging of thevirtual space 150 to therendering part 260. - The
rendering part 260 generates a two-dimensional video of thevirtual space 150 captured from thevirtual camera 130. The generated two-dimensional video is sent to the display device. For each of the other peripheral moving bodies moving relative to the virtual movingbody 100, the rendering part may display ghosts of the peripheral moving body in a discontinuous manner or display a video of the other peripheral moving body a predetermined time before the current time point in a manner superimposed on the two-dimensional image such that, for example, the farther back in time the video is from, the higher the transparency of the video is, to make the speed of the peripheral moving body relative to the virtual movingbody 100 easily recognizable. Alternatively, the rendering part may add trajectory (blurring) of the peripheral moving body to make the moving condition of the peripheral moving body recognizable. How far in the past to go back to display the ghost or video of the other moving body may be determined in advance or changed as appropriate based on an instruction of the operator (driver). - Note that ghosts (image lag), a past video, or the like may not be displayed for the host moving body, an oncoming moving body, a moving body whose absolute value of speed relative to the host moving body or absolute value of speed change relative to the host moving body is smaller than a predetermined value, a moving body whose speed is lower than the host moving body, a stationary object, or the like. Alternatively, ghosts, a past video, or the like may not be displayed for some of these moving bodies and object. Moreover, an effect of display may vary depending on the types or characteristics of the moving bodies and object.
- The
recorder part 280 stores the information on the virtual space changing from moment to moment in amemory 704 or a storage medium 760. The information stored in the recorder part is information on thevirtual space 150. Accordingly, it is possible to record information more compressed than the information of a drive recorder which records the actual traveling video. Thus, a consumed storage capacity can be reduced. Moreover, the recorded information on the virtual space changing from moment to moment allows the virtual space to be displayed on a display as a two-dimensional image with a viewpoint changed in later reproduction of the recorded information by moving the virtual camera. -
Fig. 3 is a view illustrating an example of a first display image in one embodiment. The image ofFig. 3 is an image of thevirtual space 150 captured by thevirtual camera 130. There are displayed the virtual movingbody 100 at the current time point, a peripheral movingbody 310 at the current time point, andpoles 330 present on the roadside at the current time point. - In
Fig. 2 , aghost 312, aghost 314, and aghost 316 are ghosts of the peripheral movingbody 310. - For example, the
ghost 312 may be a ghost of the peripheral moving body 310 a time d before the current time point. Moreover, theghost 312 may be an image whose transparency is increased from that of the peripheral movingbody 310 by Δ%. Accordingly, theghost 312 may be an image always the time d before the peripheral movingbody 310 with a transparency increased from that of the peripheral movingbody 310 by Δ%. In this case, when the speed of the peripheral movingbody 310 is higher than the virtual movingbody 100, theghost 312 seems to be following the peripheral movingbody 310. Note that the "transparency" described above is a term used such that transparency of 100% means a completely transparent state and transparency of 0% means a completely opaque state. - Alternatively, the
ghost 312 may be an instantaneous image of the peripheral moving body 310 a predetermined time before a certain time point with a transparency increased from that of the virtual movingbody 100 by Δ%. Specifically, the peripheral movingbody 310 may leave a ghost every predetermined interval time d. In this case, when the speed of the peripheral movingbody 310 is higher than the virtual movingbody 100, theghost 312 seems such that the peripheral movingbody 310 is moving while leaving a still image of the peripheral movingbody 100 every predetermined interval d with a transparency of the still image increased from that of the virtual movingbody 100 by Δ%. - The
ghost 314 may be an image the time d before theghost 312 with a transparency further increased from that of theghost 312 by Δ%. Theghost 316 may be an image the time d before theghost 314 with a transparency further increased from that of theghost 314 by Δ%. Note that the value by which the transparency is increased may vary among the ghosts. Moreover, the interval d of the ghost may vary among the ghosts. Note that how many ghosts are left may be determined in advance. Alternatively, the ghost may be deleted and excluded from rendering targets when the transparency of the ghost is increased and reaches or exceeds 100%. - Generating the video while leaving the ghosts of the
peripheral image 310 as described above allows the driver to know how fast the peripheral movingbody 310 is moving relative to the virtual movingbody 100. - Although description is given above of the example in which the
virtual camera 130 is moved such that the virtual movingbody 100 is stationary in the image even when the speed of the virtual movingbody 100 changes, the image may be generated such that the virtual movingbody 100 moves. - Although only one virtual moving
body 100 is illustrated in the example ofFig. 3 , multiple virtual moving bodies may be similarly displayed. Moreover, an oncoming vehicle may be displayed. The virtual movingbody 310 is not limited to a four-wheeler and may be a two-wheeler, a bicycle, or the like. - The
ghost 312, theghost 314, and theghost 316 are desirably displayed with relationships relative to the virtual movingbody 100 maintained. Thus, the ghosts may be displayed with positional relationships relative to the stationary road or roadside poles not maintained. - Note that the
stationary roadside poles 330 are desirably displayed without leaving ghosts. -
Fig. 4 is a view illustrating a modified example of the first display image in one embodiment. - A blurring
image 360 inFig. 4 is an image expressing how the peripheral movingbody 310 is moving by adding a blur effect to theghost 312, theghost 314, theghost 316, and the like. The blurring image can be expressed by various types of processing other than the processing described above as a matter of course. A video easier for the operator (driver) to understand can be provided by displaying such an effect. -
Fig. 5 is a view illustrating an example of a second display image in one embodiment.Fig. 5 illustrates an example in which a peripheral movingbody 410 is a truck and the speed thereof is lower than the virtual movingbody 100. In this case, the peripheral movingbody 410 at the current time point is displayed and aghost 412, aghost 414, and aghost 416, which are in a descending chronological order, are displayed in front of the peripheral movingbody 410. Note that when the peripheral movingbody 410 is a truck and the speed thereof is lower than the virtual movingbody 100, theghost 412, theghost 414, and theghost 416 may not be displayed. When the speed of the peripheral movingbody 410 is lower than the virtual movingbody 100 as described above, the ghosts overlap a front portion of the truck and humans sometimes have difficulty in recognizing the shape of truck (peripheral moving body 410). Thus, in such a situation, the ghosts are sometimes not displayed to allow humans to recognize the shape of the truck (peripheral moving body 410) more clearly than the moving condition of the truck. Moreover, when the absolute value of the speed of the peripheral movingbody 410 relative to the virtual movingbody 100 is smaller than a predetermined value, the overlapping of the ghosts is intense and humans sometimes have difficulty in recognizing the shape of the truck. Thus, in such a situation, the ghosts are sometimes not displayed to allow humans to recognize the shape of the truck more clearly than the moving condition of the truck. - As described above, an effect added to the display of the peripheral moving
body 100 may be changed when the speed of the peripheral movingbody 410 relative to the virtual movingbody 100 satisfies a certain condition. - Since the interval time d and the transparency Δ% are the same as those in
Fig. 3 , description thereof is omitted. -
Fig. 6 is a first flowchart of displaying the ghosts of the peripheral moving body in one embodiment. Each step is described below. - [Step S500] A processing flow of the embodiment may be started by an interruption at a predetermined interval. Note that a trigger for starting the processing is not limited to the interruption. The processing proceeds to step S510.
- [Step S510] Processing of steps between step S510 and step S526 is repeated for all peripheral objects.
- [Step S512] For example, whether an object recognized by pattern recognition based on the videos of the cameras mounted in the actual moving
body 100R is a new object is determined. When the object is a new object ("Yes"), the processing proceeds to step S514. When the object is a recognized object and is not a new object ("NO"), the processing proceeds to step S516. - [Step S514] The new object is determined and a corresponding virtual model is identified. When the object to be recognized is a new object, the vehicle type of the object or the kind of the object is determined by using techniques such as pattern recognition and machine learning and a model corresponding to the object is identified. In the pattern recognition, for example, in the case of the moving body, a vehicle type with the closest characteristics is identified based on characteristics such as the positions of the headlamps and various design features of the vehicle body. A 3D model of the identified vehicle type is obtained by searching the
3D model database 240 for the 3D model based on the identified vehicle type. Characteristics of thestationary poles 330 on the road are also extracted, the object recognition is performed for thepoles - [Step S516] The object is determined and a virtual model is identified. The virtual model is arranged at an appropriate position in the
virtual space 150 based on the position of one or more recognized objects. Next, the processing proceeds to step S518. - [Step S518] The virtual space is recorded in a recorder. When recording of the virtual space is instructed by the operator (driver) or the like, the virtual space at that time point is stored in the
memory 704 or the storage medium 760. Next, the processing proceeds to step S520. - [Step S520] The position of the model of the peripheral moving body the time d before the current time point is determined. The position in this case is desirably set such that the position relative to the virtual moving
body 100 is maintained and the positional relationship to the stationary road or theroadside poles 330 may change. The operator (driver) can thereby grasp the past positional relationship between the host virtual movingbody 100 and the other peripheral moving body. Next, the processing proceeds to step S522. - [Step S522] The position of the model of the peripheral moving body a
time 2d before the current time point is determined. The other processing is the same as that in step S520. Next, the processing proceeds to S524. - [Step S524] The position of the model of the peripheral moving body a
time 3d before the current time point is determined. The other processing is the same as that in step S520. Although only the processing for the peripheral moving body of thetime 3d before the current time point is described in the embodiment illustrated inFig. 6 , how far in the past to go back to process the model of the moving body may be determined in advance. Accordingly, the processing in the embodiment is not limited to that inFig. 6 . Next, the processing proceeds to step S526. - [Step S526] When there is any other unprocessed object, the processing returns to step S510. When there is no other unprocessed object, the processing proceeds to step S528.
- [Step S528] The two-dimensional video of the
virtual space 150 captured from thevirtual camera 130 is rendered. Note that the rendering is performed with the transparency of the model the time d before the current time point increased by Δ%, the transparency of the model thetime 2d before the current time point increased by 2Δ%, and the transparency of the model thetime 3d before the current time point increased by 3Δ% from the transparency of the peripheral moving body at the current time point, and the image illustrated inFig. 3 is thereby obtained. Although description is given of the example in which the past models are generated at even intervals d, the intervals may not be even. Moreover, instead of increasing each transparency at a predetermined ratio, the transparencies of the respective past models may be individually varied in consideration of the characteristics of the display device and the like. - By performing the aforementioned processing, as illustrated in
Fig. 3 , the peripheral movingbody 312 at a time point of the time d before the current time point is displayed to follow the image of the current peripheral movingbody 310 with the transparency thereof increased by Δ%. The past peripheral movingbody 314 and the past peripheral movingbody 316 are as described above and description thereof is thus omitted. - A video in which the peripheral moving bodies (312, 314, 316) at multiple past time points move to smoothly follow the peripheral moving
body 310 is generated by using the processing ofFig. 6 . - Performing the processing of
Fig. 6 enables display which allows the operator (driver) to more easily understand how the peripheral moving body is moving relative to the virtual movingbody 100. - Note that, in order to display blurring as in
Fig. 4 , the peripheral moving bodies (312, 314, 316) at multiple past time points may be displayed in a blurred manner. Alternatively, the past ghosts illustrated inFig. 7 may be displayed in a blurred manner. Note that expression of blurring is not limited to these methods. -
Fig. 7 is a second flowchart of displaying the ghosts of the peripheral moving body in one embodiment. Each step is described below. - [Step S600] A processing flow of the embodiment may be started by an interruption at a predetermined interval. Note that a trigger for starting the processing is not limited to the interruption. The processing proceeds to step S610.
- [Step S610] Processing of steps between step S610 and step S626 is repeated for all peripheral objects.
- [Step S612] For example, whether an object recognized by pattern recognition based on the videos of the cameras mounted in the actual moving
body 100R is a new object is determined. When the object is a new object ("Yes"), the processing proceeds to step S614. When the object is a recognized object and is not a new object, the processing proceeds to step S616. - [Step S614] The new object is determined and a corresponding virtual model is identified. When the object to be recognized is a new object, the vehicle type of the object or the kind of the object is determined by using the technique of pattern recognition and a model corresponding to the object is identified. In the pattern recognition, for example, in the case of the moving body, a vehicle type with the closest characteristics is identified based on characteristics such as the positions of the headlamps and various design features of the vehicle body. A 3D model of the identified vehicle type is obtained by searching the 3D model database for the 3D model based on the identified vehicle type. Characteristics of the
stationary poles 330 on the road are also extracted, the object recognition is performed for thepoles - [Step S616] The object is determined and a virtual model is identified. The virtual model is arranged at an appropriate position in the
virtual space 150 based on the position of one or more recognized objects. Next, the processing proceeds to step S618. - [Step S618] The virtual space is recorded in the recorder. When recording of the virtual space is instructed by the operator, the virtual space at that time point is stored in the
memory 704 or the storage medium 760. Next, the processing proceeds to step S620. - [Step S620] Whether the interval time d1 has elapsed is checked. When the interval time d1 has elapsed ("Yes"), the measurement of interval is reset to zero and the processing proceeds to step S622. When the interval time d1 has not elapsed ("No"), the processing proceeds to step S624.
- [Step S622] A rendered image of a model of the peripheral moving body at the current time point is left as a ghost. The processing proceeds to step S622. Multiple ghosts are generated one by one every interval time d1 by this step.
- [Step S624] The transparency of each of the ghosts is increased by Δ%. The ghost whose transparency reaches or exceeds 100% may be excluded from the rendering targets. The positional relationships of the ghosts to the virtual moving
body 100 are desirably maintained. Note that the ghost a predetermined time before or more may be excluded from the rendering targets even when the transparency thereof does not reach or exceed 100%. Note that the rendering may be performed at once in step S628. The processing proceeds to step S626. - [Step S626] When there is any other unprocessed object, the processing returns to step S610. When there is no other unprocessed object, the processing proceeds to step S628.
- [Step S628] The two-dimensional video of the
virtual space 150 captured from thevirtual camera 130 is rendered. Note that the rendering is performed with the transparency of the ghost the time d before the current time point increased by Δ%, the transparency of the ghost thetime 2d before the current time point increased by 2Δ%, and the transparency of the ghost thetime 3d before the current time point increased by 3Δ%, and the image illustrated inFig. 3 can be thereby obtained. Although description is given of the example in which the ghosts are generated at even intervals d1, the intervals may not be even. Moreover, instead of increasing each transparency at a predetermined ratio, the transparencies of the respective past models may be individually varied in consideration of the characteristics of the display device and the like. - By performing the aforementioned processing, as illustrated in
Fig. 3 , the image of the current peripheral movingbody 310 is displayed while leaving multiple ghosts at the intervals of time d1 with the transparency of each of the ghosts increased by Δ% from the previous one. - The ghosts (312, 314, 316) of the peripheral moving
body 312 are generated every interval time d1 by using the processing ofFig. 7 . - Performing the processing of
Fig. 7 enables display which allows the operator (driver) to more easily understand how the peripheral moving body is moving relative to the virtual movingbody 100. -
Fig. 8 is a diagram illustrating ahardware configuration 700 in one embodiment. - The
hardware configuration 700 includes a CPU 702, thememory 704, a communication control device 706, aninput interface 708, adisplay controller 710, anexternal memory controller 712, and anoutput interface 714. - Networks such as the Internet, a CAN 731 (controller area network), and the like are connected to the communication control device 706. A touch sensitive sensor 740, a GPS 742, a camera 744, and the like are connected to the
input interface 708. A display 750 is connected to thedisplay controller 710. Theexternal memory controller 712 can read data from and write data to the storage medium 760. A speaker 770 and the like are connected to the output interface. - The storage medium 760 may be a RAM, a ROM, a CD-ROM, a DVD-ROM, a hard disk, a memory card, or the like.
- A program of the embodiment described above may be executed by a computer having the
hardware configuration 700. Moreover, the program of the embodiment may be implemented as a method to be executed by a computer. Part or all of the program of the embodiment may be executed by an operating system. Moreover, part of the program may be implemented by hardware. The program may be stored in the storage medium 760 or thememory 704. - Note that the steps of the program in the aforementioned embodiment may be executed simultaneously or with the order thereof changed as long as there is no contradiction.
- The aforementioned embodiment may be implemented as a hardware device.
- The aforementioned embodiment does not limited the invention described in the claims and is treated as an example as a matter of course.
-
- 210
- moving body measurement part
- 220
- peripheral object measurement part
- 230
- object recognition part
- 240
- 3D model database
- 250
- virtual space construction part
- 260
- rendering part
- 270
- car navigation database
- 280
- recorder part
Claims (7)
- A moving body position information recognition display device comprising:a traveling state obtaining part configured to obtain information on a traveling state of a host moving body including a position and speed of the host moving body;an object information obtaining part configured to obtain information on a plurality of objects including a shape of a travel road and another moving body present in a periphery of the host moving body;a virtual space construction part configured to construct a virtual space by associating at least the host moving body and each of the plurality of objects respectively with each of a plurality of virtual models and by arranging the associated virtual models in a three-dimensional space based on the information obtained from the traveling state obtaining part and the object information obtaining part; anda two-dimensional video generation part configured to generate a two-dimensional video capturing the virtual space from a virtual camera arranged in the virtual space and output the two-dimensional video for a display screen, whereinthe two-dimensional video generation part generates the two-dimensional video while adding a predetermined effect to display of the virtual model corresponding to the object in the two-dimensional video depending on speed of the object relative to the host moving body.
- The moving body position information recognition display device according to claim 1, wherein the effect is a discretely-displayed ghost whose transparency increases over time.
- The moving body position information recognition display device according to claim 2, wherein
the ghost is a video of the virtual model at a time point a predetermined time before a current time point, and
the longer the predetermined time is, the higher the transparency of the video of the virtual model is. - The moving body position information recognition display device according to claim 1, wherein the effect is a trajectory.
- The moving body position information recognition display device according to any one of claims 1 to 4, wherein the effect is not added to the virtual model corresponding to the object coming toward the host moving body, the object whose absolute value of speed relative to the host moving body or absolute value of speed change relative to the host moving body is smaller than a predetermined value, the object whose speed is lower than the host moving body, or the object which is stationary.
- The moving body position information recognition display device according to any one of claims 1 to 5, further comprising a recording part configured to record the virtual space or the two-dimensional video.
- A moving body position information recognition display program which causes a computer to execute instructions to perform a set of operations comprising:obtaining information on a traveling state of a host moving body including a position and speed of the host moving body;obtaining information on a plurality of objects including a shape of a travel road and another moving body present in a periphery of the host moving body;constructing a virtual space by associating at least the host moving body and each of the plurality of objects respectively with each of a plurality of virtual models and by arranging the associated virtual models in a three-dimensional space based on the information obtained in the obtaining information on a traveling state of a host moving body e and the obtaining information on a plurality of objects; andgenerating a two-dimensional video capturing the virtual space from a virtual camera arranged in the virtual space and outputting the two-dimensional video for a display screen,whereinthe generating the two-dimensional video generates the two-dimensional video while adding a predetermined effect to display of the virtual model corresponding to the object in the two-dimensional video depending on speed of the object relative to the host moving body.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2017192431A JP6543313B2 (en) | 2017-10-02 | 2017-10-02 | Image generation record display device and program for mobile object |
PCT/JP2018/036758 WO2019069875A1 (en) | 2017-10-02 | 2018-10-01 | Image generation/recording/display device for moving body, and program |
Publications (2)
Publication Number | Publication Date |
---|---|
EP3693930A1 true EP3693930A1 (en) | 2020-08-12 |
EP3693930A4 EP3693930A4 (en) | 2021-06-30 |
Family
ID=65995397
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP18864949.5A Pending EP3693930A4 (en) | 2017-10-02 | 2018-10-01 | Image generation/recording/display device for moving body, and program |
Country Status (6)
Country | Link |
---|---|
US (1) | US10896542B2 (en) |
EP (1) | EP3693930A4 (en) |
JP (1) | JP6543313B2 (en) |
KR (1) | KR102349915B1 (en) |
CN (1) | CN111164652B (en) |
WO (1) | WO2019069875A1 (en) |
Families Citing this family (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2020095465A (en) * | 2018-12-12 | 2020-06-18 | ソニー株式会社 | Image processing device, image processing method, and program |
US11593581B2 (en) | 2019-02-28 | 2023-02-28 | Stats Llc | System and method for calibrating moving camera capturing broadcast video |
CN110708594B (en) * | 2019-09-26 | 2022-03-29 | 三星电子(中国)研发中心 | Content image generation method and system |
CN114520875B (en) * | 2022-01-28 | 2024-04-02 | 西安维沃软件技术有限公司 | Video processing method and device and electronic equipment |
JP7438515B2 (en) | 2022-03-15 | 2024-02-27 | オムロン株式会社 | Bird's-eye view data generation device, learning device, bird's-eye view data generation program, bird's-eye view data generation method, and robot |
WO2023176854A1 (en) * | 2022-03-15 | 2023-09-21 | オムロン株式会社 | Bird's-eye data generation device, learning device, bird's-eye data generation program, bird's-eye data generation method, and robot |
JP7364269B2 (en) | 2022-03-15 | 2023-10-18 | Necプラットフォームズ株式会社 | Object detection device, image processing display method and program |
Family Cites Families (23)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6972756B1 (en) * | 1997-11-25 | 2005-12-06 | Kabushiki Kaisha Sega Enterprises | Image generating device |
JPH11250396A (en) | 1998-02-27 | 1999-09-17 | Hitachi Ltd | Device and method for displaying vehicle position information |
WO1999049465A1 (en) * | 1998-03-20 | 1999-09-30 | Citizen Watch Co., Ltd. | Magnetic recording device |
JPH11339192A (en) * | 1998-05-25 | 1999-12-10 | Hitachi Ltd | Display device for vehicle |
JP3442736B2 (en) * | 2000-11-30 | 2003-09-02 | コナミ株式会社 | Image processing apparatus, image processing method, and information storage medium |
JP2002278662A (en) * | 2001-03-22 | 2002-09-27 | Namco Ltd | Game device and game processing program and recording medium with its program recorded |
JP2003216981A (en) | 2002-01-25 | 2003-07-31 | Iwane Kenkyusho:Kk | Automatic working system |
WO2005088970A1 (en) | 2004-03-11 | 2005-09-22 | Olympus Corporation | Image generation device, image generation method, and image generation program |
JP3769287B2 (en) * | 2004-09-09 | 2006-04-19 | コナミ株式会社 | Image generating apparatus, load display method, and program |
JP2006092156A (en) * | 2004-09-22 | 2006-04-06 | Namco Ltd | Program, information storage medium and image generation device |
JP4534922B2 (en) * | 2005-09-16 | 2010-09-01 | 株式会社デンソー | Virtual leading vehicle image display system |
JP4321509B2 (en) * | 2005-10-17 | 2009-08-26 | 株式会社デンソー | Mobile communication device |
GB0807953D0 (en) * | 2008-05-01 | 2008-06-11 | Ying Ind Ltd | Improvements in motion pictures |
JP5273068B2 (en) | 2010-02-26 | 2013-08-28 | 三菱自動車工業株式会社 | Vehicle periphery monitoring device |
JP5681569B2 (en) | 2011-05-31 | 2015-03-11 | 富士通テン株式会社 | Information processing system, server device, and in-vehicle device |
US9135754B2 (en) * | 2012-05-07 | 2015-09-15 | Honda Motor Co., Ltd. | Method to generate virtual display surfaces from video imagery of road based scenery |
US8892358B2 (en) * | 2013-03-14 | 2014-11-18 | Robert Bosch Gmbh | System and method for distortion correction in three-dimensional environment visualization |
DE102014219575A1 (en) * | 2013-09-30 | 2015-07-23 | Honda Motor Co., Ltd. | Improved 3-dimensional (3-D) navigation |
US20150104757A1 (en) * | 2013-10-15 | 2015-04-16 | Mbfarr, Llc | Driving assessment and training method and apparatus |
US10321117B2 (en) * | 2014-04-11 | 2019-06-11 | Lucasfilm Entertainment Company Ltd. | Motion-controlled body capture and reconstruction |
US9965956B2 (en) * | 2014-12-09 | 2018-05-08 | Mitsubishi Electric Corporation | Collision risk calculation device, collision risk display device, and vehicle body control device |
JP6372402B2 (en) * | 2015-03-16 | 2018-08-15 | 株式会社デンソー | Image generation device |
KR101750159B1 (en) * | 2015-07-01 | 2017-06-22 | 엘지전자 주식회사 | Assistance Apparatus for Driving of a Vehicle, Method thereof, and Vehicle having the same |
-
2017
- 2017-10-02 JP JP2017192431A patent/JP6543313B2/en active Active
-
2018
- 2018-10-01 CN CN201880064353.4A patent/CN111164652B/en active Active
- 2018-10-01 KR KR1020207012744A patent/KR102349915B1/en active IP Right Grant
- 2018-10-01 EP EP18864949.5A patent/EP3693930A4/en active Pending
- 2018-10-01 WO PCT/JP2018/036758 patent/WO2019069875A1/en unknown
-
2020
- 2020-04-01 US US16/837,981 patent/US10896542B2/en active Active
Also Published As
Publication number | Publication date |
---|---|
CN111164652B (en) | 2023-12-08 |
US10896542B2 (en) | 2021-01-19 |
EP3693930A4 (en) | 2021-06-30 |
CN111164652A (en) | 2020-05-15 |
JP2019067174A (en) | 2019-04-25 |
US20200234497A1 (en) | 2020-07-23 |
KR20200054322A (en) | 2020-05-19 |
KR102349915B1 (en) | 2022-01-10 |
JP6543313B2 (en) | 2019-07-10 |
WO2019069875A1 (en) | 2019-04-11 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP3693930A1 (en) | Image generation/recording/display device for moving body, and program | |
US11657604B2 (en) | Systems and methods for estimating future paths | |
US11669102B2 (en) | Navigating a vehicle based on a detected barrier | |
US11392135B2 (en) | Systems and methods for navigating lane merges and lane splits | |
CN109215433B (en) | Vision-based driving scenario generator for automated driving simulation | |
US10489686B2 (en) | Object detection for an autonomous vehicle | |
JP7120820B2 (en) | VEHICLE CONTROL DEVICE, VEHICLE CONTROL METHOD, AND PROGRAM | |
JP2022523084A (en) | Prediction of 3D features for autonomous driving | |
EP4378783A1 (en) | Automatic prediction and altruistic response to a vehicle cutting in a lane | |
EP3407249A2 (en) | A method and system for generating and using a perception scene graph in motor vehicle applications | |
EP3419001A1 (en) | Method of using a perception information and system for generating a perception scene graph having a focus region for a motor vehicle | |
JP6697115B2 (en) | Image generating / recording / displaying device for moving body and program | |
US20200120273A1 (en) | Overlooking image generation system for vehicle and method thereof | |
US20220053124A1 (en) | System and method for processing information from a rotatable camera | |
JP7367709B2 (en) | Information processing device, information processing system, and information processing program |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20200422 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
DAV | Request for validation of the european patent (deleted) | ||
DAX | Request for extension of the european patent (deleted) | ||
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R079 Free format text: PREVIOUS MAIN CLASS: G06T0013200000 Ipc: G06T0015200000 |
|
A4 | Supplementary search report drawn up and despatched |
Effective date: 20210531 |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G06T 15/20 20110101AFI20210525BHEP Ipc: G06T 13/20 20110101ALI20210525BHEP Ipc: G06T 19/00 20110101ALI20210525BHEP Ipc: G07C 5/00 20060101ALI20210525BHEP Ipc: H04N 7/18 20060101ALI20210525BHEP Ipc: G08G 1/16 20060101ALI20210525BHEP |
|
RAP1 | Party data changed (applicant data changed or rights of an application transferred) |
Owner name: KAGA FEI CO., LTD. |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
17Q | First examination report despatched |
Effective date: 20240319 |