KR102677983B1 - Method for determining failure of a projector installed on autonomous vehicle and projector fault determining device using the same - Google Patents

Abstract

A method of determining a failure of a projector installed in an autonomous vehicle, comprising: (a) causing a projector failure diagnosis device to generate specific content corresponding to the state of the autonomous vehicle and transmit it to the projector; and (b) when it is determined that the autonomous vehicle is driving, the projector failure diagnosis device (b1) causes the projector to project the specific content on the floor, and causes the camera to project the content projected on the floor. acquire a content image by shooting specific content, (b2) cause a deep learning network to perform an operation on the content image, and cause an image processing controller to obtain information about the specific content obtained from the projector controller and A method including determining whether the projector is broken by referring to the decision information output by the deep learning network is disclosed.

Description

Method for determining failure of a projector installed in an autonomous vehicle and a projector failure diagnosis device using the same {METHOD FOR DETERMINING FAILURE OF A PROJECTOR INSTALLED ON AUTONOMOUS VEHICLE AND PROJECTOR FAULT DETERMINING DEVICE USING THE SAME}

This is about a method of determining a malfunction of a projector installed in an autonomous vehicle and a projector malfunction diagnosis device using the same.

As self-driving vehicles are commercialized, the importance of technology related to interaction between self-driving vehicles and surrounding vehicles and road users is emerging to ensure the safety of surrounding vehicles and road users.

Research is being conducted in various fields such as autonomous vehicle driving control, communication, security, and infrastructure, but V2H (Vehicle to Human) technology is a technology that can ensure the safety of drivers and road users and improve the acceptability of autonomous vehicles. This is being studied.

V2H communication technology refers to technology for intuitive communication between road users and autonomous vehicles, and may include ICT technologies such as V2X in addition to visual and auditory elements.

In particular, for V2H communication, methods are being used to install an external display display so that surrounding vehicles and road users can recognize the current status of the autonomous vehicle, or to project various contents on the road using a projector.

The road projector projects various contents onto the floor through a beam projector with built-in headlights, and can provide information about autonomous driving, whether the target object is recognized, information on left and right turn attempts, and whether the driver is in the autonomous vehicle. However, there is a problem in that there is no way to quickly and easily determine if the projector is broken.

Therefore, the present invention proposes a method for determining the failure of a projector installed in an autonomous vehicle when the autonomous vehicle is driving and when the vehicle is stopped.

The present invention aims to solve all of the above-mentioned problems.

In addition, the present invention, when it is determined that the autonomous vehicle is driving, causes the projector to project specific content on the floor, has the camera capture the specific content projected on the floor to obtain a content image, and uses a deep The learning network performs an operation on the content image, and the image processing controller determines whether the projector is broken by referring to the information about the specific content obtained from the projector controller and the judgment information output by the deep learning network. Another purpose is to enable judgment.

In addition, the present invention, when it is determined that the autonomous vehicle is stopped, before specific content is projected on the floor by the projector, the camera produces a pre-image before projection on the floor and a post-image after projection on the floor. Another purpose is to obtain additional images, have the image processing controller calculate the difference image between the pre-image and the after-image, and additionally compare the difference image and specific content to determine whether the projector is broken.

The characteristic configuration of the present invention for achieving the purpose of the present invention as described above and realizing the characteristic effects of the present invention described later is as follows.

According to one aspect of the present invention, in the method of determining a failure of a projector installed in an autonomous vehicle, (a) a projector failure diagnosis device causes a projector controller to generate specific content corresponding to the state of the autonomous vehicle, transmitting data to the projector; and (b) when it is determined that the autonomous vehicle is driving, the projector failure diagnosis device (b1) causes the projector to project the specific content on the floor, and causes the camera to project the content projected on the floor. acquire a content image by shooting specific content, (b2) have a deep learning network perform an operation on the content image, and cause an image processing controller to obtain information about the specific content obtained from the projector controller and A method including determining whether the projector is broken by referring to the decision information output by the deep learning network is disclosed.

As an example, before step (b2), (b11) the projector failure diagnosis device generates a target transformation area by causing a pre-processing controller to apply homography coordinate transformation to at least some target areas of the acquired content images. step; (b12) generating, by the projector failure diagnosis device, a converted cropped image by causing the preprocessing controller to crop the target converted area; and (b13) allowing the preprocessing controller to input the converted crop image to the deep learning network, by the projector failure diagnosis device.

As an example, in step (b11), the homography coordinate transformation defines the relationship between the target plane corresponding to the floor and the virtual plane corresponding to the camera when projecting the specific content on the floor. A method characterized in that it is performed using an H matrix is disclosed.

As an example, in step (b2), (b21) the projector failure diagnosis device causes the deep learning network to apply at least one convolution operation to the transformed crop image to generate a feature map, and the feature map Applying a fully connected operation to output classification information including a class score; and (b22) the projector failure diagnosis device causes the image processing controller to additionally refer to the information about the specific content obtained from the projector controller and the classification information output by the deep learning network. A method is disclosed that further includes a step of determining whether the projector is broken.

As an example, the projector failure diagnosis device causes the image processing controller to project the judgment information onto a grid cell and compare the information projected on the grid cell with information about the specific content to detect a malfunction area. A method is disclosed, characterized in that it determines whether the projector is broken.

As an example, a method is disclosed wherein the judgment information includes map data projected based on the grid cells, and the map data includes defect scores for each grid.

As an example, in step (b), if it is determined that the autonomous vehicle is stopped, the projector failure diagnosis device, (b3), sends the camera to the camera before the specific content is projected on the floor by the projector. additionally acquire a pre-image before projection on the floor and a post-image after projection on the floor, (b4) cause the image processing controller to calculate a difference image between the pre-image and the post-image; A method is disclosed that further includes the step of additionally comparing the difference image and the specific content to determine whether the projector is broken.

As an example, the projector failure diagnosis device causes the image processing controller to project information about the difference image onto a grid cell and compare the information projected onto the grid cell with information about the specific content to detect a malfunction area. A method is disclosed that is characterized in that it determines whether the projector is broken through this.

As an example, the specific content includes content indicating whether the autonomous vehicle is driving, content indicating actions to be performed by the autonomous vehicle within a preset time, and content indicating whether the autonomous vehicle is close to a pedestrian. A method is disclosed, characterized in that it includes text information corresponding to at least part of the text.

As an example, a method is disclosed wherein the position of the projector is installed ahead of the position of the camera, and the height of the projector is installed below the height of the camera.

According to another aspect of the present invention, a projector failure diagnosis device for determining a failure of a projector installed in an autonomous vehicle, comprising: at least one memory storing instructions; and at least one processor configured to execute the instructions, wherein the processor includes: (I) a process for causing a projector controller to generate specific content corresponding to a state of the autonomous vehicle and transmit it to the projector; and (II) when it is determined that the autonomous vehicle is driving, (II-1) cause the projector to project the specific content on the floor, and cause the camera to photograph the specific content projected on the floor. Obtain a content image, (II-2) have a deep learning network perform an operation on the content image, and have an image processing controller obtain information about the specific content obtained from the projector controller and the deep learning A projector failure diagnosis apparatus is disclosed that performs a process of determining whether the projector is broken by referring to decision information output by a network.

As an example, before the process (II-2), the processor (II-11) causes the preprocessing controller to apply homography coordinate transformation to at least some target areas of the acquired content images to generate a target transformation area. process; (II-12) A process of generating a converted cropped image by causing the preprocessing controller to crop the target converted area, and (II-13) having the preprocessing controller input the converted cropped image to the deep learning network. A projector failure diagnosis device is disclosed, characterized in that it further performs a process for doing so.

As an example, in the process (II-11), the homography coordinate transformation determines the relationship between the target plane corresponding to the floor and the virtual plane corresponding to the camera when projecting the specific content to the floor. A projector failure diagnosis device is disclosed, which is characterized in that it is performed using a stipulated H matrix.

As an example, in the process (II-2), the processor (II-21) causes the deep learning network to apply at least one convolution operation to the transformed crop image to generate a feature map, and A process for applying fully connected operations to a map to output classification information including class scores; and (II-22) by causing the image processing controller to additionally refer to the information about the specific content obtained from the projector controller and the classification information output by the deep learning network to detect a failure of the projector. A projector failure diagnosis device is disclosed, characterized in that it further performs a process to determine whether or not the projector failure is present.

As an example, the processor causes the image processing controller to project the determination information onto a grid cell, compare the information projected on the grid cell with information about the specific content, detect a malfunction area, and thereby detect the malfunction area of the projector. A projector failure diagnosis device is disclosed, which is characterized in that it determines whether there is a failure.

As an example, a projector failure diagnosis apparatus is disclosed, wherein the judgment information includes map data projected based on the grid cells, and the map data includes defect scores for each grid.

As an example, in the process (II), if it is determined that the autonomous vehicle is stopped, the processor causes the camera to be activated before the specific content is projected on the floor by the projector (II-3). Additionally obtain a pre-image before projection on the floor and a post-image after projection on the floor, and (II-4) cause the image processing controller to calculate a difference image between the pre-image and the post-image. A projector failure diagnosis apparatus is disclosed, which further performs a process of additionally comparing the difference image and the specific content to determine whether the projector is broken.

As an example, the processor causes the image processing controller to project information about the difference image onto a grid cell and compare the information projected onto the grid cell with information about the specific content to detect a malfunction area. A projector failure diagnosis device is disclosed, which is characterized in that it determines whether the projector is broken.

As an example, the specific content includes content indicating whether the autonomous vehicle is driving, content indicating actions to be performed by the autonomous vehicle within a preset time, and content indicating whether the autonomous vehicle is close to a pedestrian. A projector failure diagnosis device is disclosed, which includes at least part of corresponding text information.

As an example, a projector failure diagnosis device is disclosed, wherein the position of the projector is installed ahead of the position of the camera, and the height of the projector is installed below the height of the camera.

According to the present invention, the following effects are achieved.

In the present invention, when it is determined that an autonomous vehicle is driving, a projector projects specific content on the floor, a camera acquires a content image by filming specific content projected on the floor, and a deep learning network perform an operation on the content image, and have the image processing controller determine whether the projector is broken by referring to the information about the specific content obtained from the projector controller and the judgment information output by the deep learning network. It has the effect of allowing you to do so.

In addition, the present invention, when it is determined that the autonomous vehicle is stopped, before specific content is projected on the floor by the projector, the camera produces a pre-image before projection on the floor and a post-image after projection on the floor. This has the effect of allowing additional acquisition, allowing the image processing controller to calculate the difference image between the pre-image and the after-image, and additionally comparing the difference image and specific content to determine whether the projector is broken.

FIG. 1 is a diagram illustrating a schematic configuration of a projector failure diagnosis device for determining a failure of a projector installed in an autonomous vehicle according to an embodiment of the invention.
FIG. 2 is a diagram showing a specific configuration included in a projector failure diagnosis device for determining a failure of a projector installed in an autonomous vehicle according to an embodiment of the present invention.
FIG. 3 is a diagram illustrating a schematic sequence of a method for determining a failure of a projector installed in an autonomous vehicle according to an embodiment of the present invention.
FIG. 4 is a diagram illustrating an example showing the positions and heights of a projector and camera installed in an autonomous vehicle according to an embodiment of the present invention.
Figure 5 is a schematic sequence of inputting a crop image to a deep learning network to determine a failure of a projector installed in the autonomous vehicle when it is determined that the autonomous vehicle is driving according to an embodiment of the present invention. This is a drawing for explanation.
Figure 6 illustrates an example of generating a target conversion area and generating a crop image to determine a failure of a projector installed in the autonomous vehicle when it is determined that the autonomous vehicle is driving according to an embodiment of the present invention. This is a drawing for
FIG. 7 is a diagram illustrating additional steps of a method for determining a failure of a projector installed in an autonomous vehicle when it is determined that the autonomous vehicle is driving according to an embodiment of the present invention.
FIG. 8 is a diagram illustrating an example of map data and defect scores for determining a failure of a projector installed in the autonomous vehicle when it is determined that the autonomous vehicle is driving according to an embodiment of the present invention.
FIG. 9 is a diagram illustrating a schematic sequence of a method for determining a failure of a projector installed in an autonomous vehicle when it is determined that the autonomous vehicle is stopped according to an embodiment of the present invention.
Figure 10 illustrates an example of a difference image between a pre-image and a post-image for determining a failure of a projector installed in the autonomous vehicle when it is determined that the autonomous vehicle is stopped according to an embodiment of the present invention. It is a drawing.
FIG. 11 illustrates an example of homography coordinate transformation performed using the H matrix to determine failure of a projector installed in the autonomous vehicle when it is determined that the autonomous vehicle is driving according to an embodiment of the present invention. This is a drawing for this purpose.

The detailed description of the present invention described below refers to the accompanying drawings, which show by way of example specific embodiments in which the present invention may be practiced to make clear the objectives, technical solutions and advantages of the present invention. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention.

Additionally, throughout the description and claims, the word “comprise” and variations thereof are not intended to exclude other technical features, attachments, components or steps. Other objects, advantages and features of the invention will appear to those skilled in the art, partly from this description and partly from practice of the invention. The examples and drawings below are provided by way of example and are not intended to limit the invention.

Moreover, the present invention encompasses all possible combinations of the embodiments shown herein. It should be understood that the various embodiments of the invention are different from one another but are not necessarily mutually exclusive. For example, specific shapes, structures and characteristics described herein with respect to one embodiment may be implemented in other embodiments without departing from the spirit and scope of the invention. Additionally, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the detailed description that follows is not intended to be taken in a limiting sense, and the scope of the invention is limited only by the appended claims, together with all equivalents to what those claims assert, if properly described. Similar reference numbers in the drawings refer to identical or similar functions across various aspects.

Hereinafter, in order to enable those skilled in the art to easily practice the present invention, preferred embodiments of the present invention will be described in detail with reference to the attached drawings.

FIG. 1 is a diagram illustrating a schematic configuration of a projector failure diagnosis device 100 that determines a failure of a projector installed in an autonomous vehicle according to an embodiment of the invention.

Referring to FIG. 1, the projector failure diagnosis device 100 includes a memory 110 that stores instructions for determining a failure of a projector installed in an autonomous vehicle, and a processor that detects a failure of the projector in response to the instructions stored in the memory. It may include (120). At this time, the projector failure diagnosis device 100 may include a personal computer (PC), a mobile computer, etc.

Specifically, the projector fault diagnosis device 100 is typically a computing device (e.g., a device that may include computer processors, memory, storage, input devices and output devices, and other components of conventional computing devices; routers, switches, etc. using a combination of computer software (i.e., instructions that enable a computing device to function in a certain way) (e.g., electronic communication devices; electronic information storage systems, such as network attached storage (NAS) and storage area networks (SAN)); This may be to achieve the desired system performance.

Additionally, the processor of the projector failure diagnosis device 100 may include hardware components such as a Micro Processing Unit (MPU) or Central Processing Unit (CPU), cache memory, and data bus. Additionally, the projector failure diagnosis device 100 may further include an operating system and software configuration of an application that performs a specific purpose.

However, the case where the projector failure diagnosis device 100 includes an integrated processor in which a medium, a processor, and a memory are integrated for implementing the present invention is not excluded.

Additionally, the projector failure diagnosis device 100 may be linked to a database (not shown) containing information used to determine failure of a projector installed in an autonomous vehicle. Here, the database (not shown) is a flash memory type, hard disk type, multimedia card micro type, or card type memory (for example, SD or XD memory). , RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (ReadOnly Memory, ROM), EEPROM (Electrically Erasable Programmable ReadOnly Memory), PROM (Programmable ReadOnly Memory), magnetic memory, magnetic disk, and optical disk. It may include at least one type of storage medium, but is not limited to this and may include any medium capable of storing data. In addition, the database (not shown) may be installed separately from the projector failure diagnosis device 100, or alternatively, may be installed inside the projector failure diagnosis device 100 to transmit data or record received data, or may be installed separately from the projector failure diagnosis device 100. It may be implemented separately, and this may vary depending on the implementation conditions of the invention.

In addition, the projector failure diagnosis device 100 is located inside the autonomous vehicle and may transmit and receive information sensed by a plurality of sensors mounted on the autonomous vehicle, and may communicate with a server located outside the autonomous vehicle and the necessary Information may also be sent and received. Here, the plurality of sensors may include, but are not limited to, a lidar sensor, a radar sensor, a camera sensor, and a GPS sensor.

Additionally, the specific configuration of the projector failure diagnosis device 100 is as follows.

FIG. 2 is a diagram showing a specific configuration included in the projector failure diagnosis device 100 for determining a failure of a projector installed in an autonomous vehicle according to an embodiment of the present invention.

The projector failure diagnosis device 100 may further include a projector 130, a camera 140, a projector controller 150, an image processing controller 160, a preprocessing controller 170, and a deep learning network 180. . However, it is not limited to this, and at least some of them may be located outside the projector failure diagnosis device 100 and configured to be interlocked with the projector trouble diagnosis device 100. In FIG. 2 , for convenience of explanation, the memory 110 and processor 120 mentioned in FIG. 1 are omitted.

The projector controller 150 can generate specific content corresponding to the state of the autonomous vehicle and transmit it to the projector 130, the projector 130 can project the specific content onto the road surface, and the camera 140 A content image can be obtained by photographing specific content projected on the floor.

In addition, the deep learning network 180 can perform operations on content images, and the image processing controller 160 outputs information about specific content obtained from the projector controller 150 and the deep learning network 180. It is possible to determine whether the projector 130 is broken by referring to the determined judgment information.

Additionally, the preprocessing controller 170 may apply homography coordinate transformation to the content image and then crop a predetermined transformation area to generate a transformed crop image, and may also input the transformed cropped image to the deep learning network 180.

This will be explained in more detail below.

A method of determining a failure of a projector installed in an autonomous vehicle using the projector failure diagnosis device 100 according to an embodiment of the present invention configured as described above will be described with reference to FIG. 3 as follows.

FIG. 3 is a diagram illustrating a schematic sequence of a method for determining a failure of a projector installed in an autonomous vehicle according to an embodiment of the present invention.

First, the projector failure diagnosis device 100 can cause the projector controller 150 to generate specific content corresponding to the state of the autonomous vehicle and transmit it to the projector 130 (S210).

Here, the specific content corresponds to at least some of the following: content indicating the driving status of the autonomous vehicle, content indicating actions to be performed by the autonomous vehicle within a preset time, and content indicating whether the autonomous vehicle is close to a pedestrian. Can contain text information.

For example, information about whether the autonomous vehicle is driving or stopped can be displayed in text, information about whether the autonomous vehicle will turn left or right can be displayed in text, and information about whether the autonomous vehicle will turn left or right can be displayed in text. You can indicate in text how close you are.

For reference, auditory elements (e.g., alarms) may also be displayed along with text.

Meanwhile, when it is determined that the autonomous vehicle is driving (S220), the projector failure diagnosis device 100 causes the projector 130 to project specific content on the floor and causes the camera 140 to project the content on the floor. A content image can be acquired by shooting specific content (S221), and then the deep learning network 180 is made to perform an operation on the content image, and the image processing controller 160 is used to operate the projector controller 150. It is possible to determine whether the projector is broken by referring to the information about the specific content obtained from and the judgment information output by the deep learning network 180 (S222). Here, the judgment information may be classification information of the deep learning network 180, which will be described later, but may also be a defect score according to information on the feature map.

This will be explained in more detail in Figures 5, 6, and 8, which will be described later.

Meanwhile, the location and height where the projector 130 and camera 140 are installed can be described as follows with reference to FIG. 4.

FIG. 4 is a diagram illustrating an example showing the positions and heights of the projector 130 and camera 140 installed in an autonomous vehicle according to an embodiment of the present invention.

As shown in Figure 4, the position of the projector 130 is characterized in that it is installed in front of the position of the camera 140, and the height of the projector 130 is characterized in that it is installed below the height of the camera 140. Do this.

Next, Figure 5 shows a crop image using the deep learning network 180 to determine a failure of the projector 130 installed in the autonomous vehicle when it is determined that the autonomous vehicle is driving according to an embodiment of the present invention. This is a diagram to roughly explain the steps required to input .

First, the projector failure diagnosis device 100 causes the preprocessing controller 170 to apply homography coordinate transformation to at least some target areas included in the content image acquired by the camera 140 to generate a target transformation area. (S210-1).

For reference, homography coordinate transformation uses an H matrix that specifies the relationship between the target plane corresponding to the floor (i.e., ground) and the virtual plane corresponding to the camera 140 when projecting specific content onto the floor. It can be performed by doing this.

This is explained with reference to FIG. 11 as follows.

Figure 11 shows a homography coordinate transformation performed using the H matrix to determine a failure of the projector 130 installed in the autonomous vehicle when it is determined that the autonomous vehicle is driving according to an embodiment of the present invention. This is a drawing to explain an example.

As shown in FIG. 11, the coordinates (P1, P2, P3, P4, P5) of the target plane corresponding to the floor surface (i.e., the ground) and the coordinates of the virtual plane (P1', P2', P3', P4', P5') can be calculated through the homography function.

Thereafter, the projector failure diagnosis apparatus 100 may generate a converted cropped image by causing the preprocessing controller 170 to crop the target converted area (S210-2).

Next, the projector failure diagnosis device 100 may cause the preprocessing controller 170 to input the converted crop image to the deep learning network 180 (S210-3).

This is explained with reference to FIG. 6 as follows.

Figure 6 is an example of generating a target conversion area and generating a crop image to determine a failure of the projector 130 installed in the autonomous vehicle when it is determined that the autonomous vehicle is driving according to an embodiment of the present invention. This is a drawing to explain.

As shown in FIG. 6, the preprocessing controller 170 can be configured to acquire the content image 610. The content image 610 includes specific content 611 such as text, and the content image 610 including the text in an obliquely output state may be obtained. It is assumed that the specific content 611 in FIG. 6 is 'autonomous driving'. In other words, surrounding vehicles and road users are informed that the autonomous vehicle is currently driving autonomously through the specific content 611. Thereafter, the preprocessing controller 170 may apply homography coordinate transformation to at least some target areas of the content image 610 containing text output at an angle to generate the target transformation area 620. In addition, the preprocessing controller 170 can be configured to crop the target conversion area 620 to generate a converted cropped image 630.

Afterwards, the preprocessing controller 170 can be configured to input the converted cropped image 630 generated in this way to the deep learning network 180.

This will be explained in more detail in FIGS. 7 and 8.

FIG. 7 is a diagram illustrating additional steps of a method for determining a failure of a projector installed in an autonomous vehicle when it is determined that the autonomous vehicle is driving according to an embodiment of the present invention.

First, the projector failure diagnosis device 100 causes the deep learning network 180 to generate a feature map by applying at least one convolution operation to the converted crop image 630, and performs a fully connected operation on the feature map. By applying it, classification information including class scores can be output (S220-1).

Thereafter, the projector failure diagnosis device 100 causes the image processing controller 160 to use information about specific content obtained from the projector controller 150 and the classification information output by the deep learning network 180. For additional reference, it is possible to determine whether the projector 130 is broken (S220-2).

Here, the projector failure diagnosis device 100 causes the image processing controller 160 to project the judgment information of the deep learning network 180 onto the grid cell and compare the information projected on the grid cell with information about specific content. It is also possible to detect malfunction areas and determine whether the projector is broken.

Here, the judgment information includes map data projected based on grid cells, and the map data may include defect scores for each grid.

This is explained with reference to FIG. 8 as follows.

FIG. 8 illustrates an example of map data and defect scores for determining a failure of the projector 130 installed in the autonomous vehicle when it is determined that the autonomous vehicle is driving according to an embodiment of the present invention. It is a drawing.

As shown in FIG. 8, a feature map is generated by applying at least one convolution operation to the converted crop image 630. Afterwards, fully connected operation is applied to the feature map to output classification information including class scores.

Here, the judgment information of the deep learning network 180 may include clashification information, and the clashification information determines how the text included in the converted crop image 630 is likely to be recognized as shown in FIG. 8. It may mean the probability value for multiple candidate groups. For example, it was assumed that the text corresponding to the class score with the highest probability was the text 'autonomous driving'.

In addition, the judgment information of the deep learning network 180 may include the defect score for each grid for the feature map as shown in FIG. 8, and the malfunction area 810 is detected by reflecting the defect score for each grid, It is also possible to determine whether the projector 130 is broken.

FIG. 9 is a diagram illustrating a schematic sequence of a method for determining a failure of a projector installed in an autonomous vehicle when it is determined that the autonomous vehicle is stopped according to an embodiment of the present invention.

S210 described above can be performed in the same way.

That is, the projector failure diagnosis device 100 can cause the projector controller 150 to generate specific content corresponding to the state of the autonomous vehicle and transmit it to the projector 130 (S210).

Thereafter, when it is determined that the autonomous vehicle is stopped (S220'), the projector failure diagnosis device 100 causes the camera 140 to project the specific content on the floor before it is projected on the floor by the projector 130. A pre-image can be acquired before the content is projected on the floor, and the camera 140 can be used to additionally acquire a post-image projected on the floor after the specific content is projected on the floor by the projector 130 (S221').

Afterwards, the image processing controller 160 can calculate the difference image between the pre-image and the after-image, and additionally compare the difference image and specific content to determine whether the projector 130 is broken (S222').

Here, the projector failure diagnosis device 100 causes the image processing controller 160 to project information about the difference image onto a grid cell and compare the information projected on the grid cell with information about specific content to detect a malfunction area. And through this, it is determined whether the projector 130 is broken.

For reference, even when it is determined that the autonomous vehicle is stopped, S210-1 and S210-2 described above may be performed.

In other words, when the autonomous vehicle is stopped, it is possible to determine whether the projector 130 is broken by calculating the difference image between the pre-image and the after-image and comparing the difference image and specific content. However, as mentioned earlier, Similar steps can be performed when an autonomous vehicle is driving.

In other words, the projector failure diagnosis apparatus 100 may generate a target transformation area by having the preprocessing controller 170 apply homography coordinate transformation to at least some target areas among the acquired content images (S210-1). .

Additionally, the projector failure diagnosis apparatus 100 may generate a converted cropped image by causing the preprocessing controller 170 to crop the target converted area (S210-2).

That is, a target transformation area is created through homography coordinate transformation, a transformation crop image is generated by cropping the target transformation area, a difference image is obtained through this, and then it is possible to determine whether the projector 130 is broken.

This is explained with reference to FIG. 10 as follows.

Figure 10 illustrates an example of a difference image between a pre-image and a post-image for determining a failure of a projector installed in the autonomous vehicle when the autonomous vehicle is determined to be stopped according to an embodiment of the present invention. It is a drawing.

As shown in Figure 10, the camera 140 is used to acquire a pre-image 1010 before specific content is projected on the floor. Afterwards, a post-image 1020 after specific content was projected onto the floor was additionally acquired.

Based on this, the difference image 1030 between the pre-image and the after-image was obtained, and the difference image and specific content (i.e., 'autonomous driving' text) were compared to determine whether the projector 130 was broken.

Likewise, the malfunction area may be detected by projecting information about the difference image 1030 onto a grid cell and comparing it with specific content.

Embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., singly or in combination. The program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention or may be known and usable by those skilled in the computer software field. Examples of computer-readable recording media include hard disks, magnetic media such as floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magnetooptical media such as floptical disks. , and hardware devices specifically configured to store and perform program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include not only machine language code such as that created by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform processing according to the invention and vice versa.

In the above, the present invention has been described with specific details such as specific components and limited embodiments and drawings, but this is only provided to facilitate a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , a person skilled in the art to which the present invention pertains can make various modifications and variations from this description.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and the scope of the patent claims described below as well as all modifications equivalent to or equivalent to the scope of the claims fall within the scope of the spirit of the present invention. They will say they do it.

Claims (20)

In a method of determining a malfunction of a projector installed in an autonomous vehicle,
(a) a projector failure diagnosis device causing a projector controller to generate specific content corresponding to the state of the autonomous vehicle and transmit it to the projector; and
(b) When it is determined that the autonomous vehicle is driving, the projector failure diagnosis device (b1) causes the projector to project the specific content on the floor, and causes the camera to project the specific content projected on the floor. acquire a content image by shooting content, (b2) cause a deep learning network to perform an operation on the content image, and cause an image processing controller to obtain information about the specific content obtained from the projector controller and the Determining whether the projector is broken by referring to decision information output by a deep learning network;
Including,
Before step (b2) above,
(b11) generating, by the projector failure diagnosis apparatus, a target transformation area by having a pre-processing controller apply homography coordinate transformation to at least some target areas of the acquired content images;
(b12) generating, by the projector failure diagnosis device, a converted cropped image by causing the preprocessing controller to crop the target converted area; and
(b13) allowing the projector failure diagnosis device to input the converted crop image to the deep learning network by the preprocessing controller;
Characterized by further comprising,
In step (b2) above,
(b21) The projector failure diagnosis device causes the deep learning network to generate a feature map by applying at least one convolution operation to the transformed crop image, and applies a fully connected operation to the feature map to generate a class score. A step of outputting conflict information including; and
(b22) The projector failure diagnosis device causes the image processing controller to additionally refer to the information about the specific content obtained from the projector controller and the classification information output by the deep learning network, determining whether the projector is broken;
A method further comprising: delete According to paragraph 1,
In step (b11) above,
The homography coordinate transformation is performed using an H matrix that defines the relationship between a target plane corresponding to the floor and a virtual plane corresponding to the camera when projecting the specific content on the floor. How to. delete According to paragraph 1,
The projector failure diagnosis device causes the image processing controller to project the determination information onto a grid cell, compare the information projected on the grid cell with information about the specific content, detect a malfunction area, and thereby detect the malfunction area of the projector. A method characterized by determining whether there is a failure. According to clause 5,
The decision information includes map data projected based on the grid cells, and the map data includes defect scores for each grid. In a method of determining a malfunction of a projector installed in an autonomous vehicle,
(a) a projector failure diagnosis device causing a projector controller to generate specific content corresponding to the state of the autonomous vehicle and transmit it to the projector; and
(b) When it is determined that the autonomous vehicle is driving, the projector failure diagnosis device (b1) causes the projector to project the specific content on the floor, and causes the camera to project the specific content projected on the floor. acquire a content image by shooting content, (b2) cause a deep learning network to perform an operation on the content image, and cause an image processing controller to obtain information about the specific content obtained from the projector controller and the Determining whether the projector is broken by referring to decision information output by a deep learning network;
Including,
In step (b) above,
When it is determined that the autonomous vehicle is stopped, the projector failure diagnosis device (b3) displays a pre-image before the specific content is projected on the floor by the camera, before being projected on the floor by the camera. and additionally acquire a post-image after projection on the floor, (b4) cause the image processing controller to calculate a difference image between the pre-image and the post-image, and additionally obtain the difference image and the specific content. The method further includes comparing to determine whether the projector is broken. In clause 7,
The projector failure diagnosis device causes the image processing controller to project information about the difference image onto a grid cell and compare the information projected onto the grid cell with information about the specific content to detect a malfunction area. A method characterized by determining whether the projector is broken. According to paragraph 1,
The specific content includes at least some of content indicating whether the autonomous vehicle is driving, content indicating actions to be performed by the autonomous vehicle within a preset time, and content indicating whether the autonomous vehicle is close to a pedestrian. A method characterized by including corresponding text information. According to paragraph 1,
The method is characterized in that the position of the projector is installed in front of the position of the camera, and the height of the projector is installed below the height of the camera. In a projector failure diagnosis device that determines a failure of a projector installed in an autonomous vehicle,
at least one memory storing instructions; and
at least one processor configured to execute the instructions;
Including,
The processor may include: (I) a process for causing a projector controller to generate specific content corresponding to the state of the autonomous vehicle and transmit it to the projector; and (II) when it is determined that the autonomous vehicle is driving, (II-1) cause the projector to project the specific content on the floor, and cause the camera to photograph the specific content projected on the floor. Obtain a content image, (II-2) have a deep learning network perform an operation on the content image, and have an image processing controller obtain information about the specific content obtained from the projector controller and the deep learning A process of determining whether the projector is broken, with reference to the judgment information output by the network,
Before the process (II-2) above,
The processor may include: (II-11) a process of causing a pre-processing controller to generate a target transformation area by applying homography coordinate transformation to at least some target areas of the acquired content images; (II-12) A process of generating a converted cropped image by causing the preprocessing controller to crop the target converted area, and (II-13) having the preprocessing controller input the converted cropped image to the deep learning network. Characterized by further performing a process to do so,
In the process (II-2) above,
The processor (II-21) causes the deep learning network to generate a feature map by applying at least one convolution operation to the transformed crop image, and applies a fully connected operation to the feature map to generate a class score. A process for outputting classification information including; and (II-22) by causing the image processing controller to additionally refer to the information about the specific content obtained from the projector controller and the classification information output by the deep learning network to detect a failure of the projector. A projector failure diagnosis device characterized in that it further performs a process to determine whether or not the projector failure is present. delete According to clause 11,
In the process (II-11) above,
The homography coordinate transformation is characterized by being performed using an H matrix that defines the relationship between a target plane corresponding to the floor and a virtual plane corresponding to the camera when projecting the specific content on the floor. A projector failure diagnosis device. delete According to clause 11,
The processor causes the image processing controller to project the determination information onto a grid cell, compare the information projected on the grid cell with information about the specific content, detect a malfunction area, and determine whether the projector is broken through this. A projector failure diagnosis device characterized by making a decision. According to clause 15,
The judgment information includes map data projected based on the grid cells, and the map data includes a defect score for each grid. In a projector failure diagnosis device that determines a failure of a projector installed in an autonomous vehicle,
at least one memory storing instructions; and
at least one processor configured to execute the instructions;
Including,
The processor may include: (I) a process for causing a projector controller to generate specific content corresponding to the state of the autonomous vehicle and transmit it to the projector; and (II) when it is determined that the autonomous vehicle is driving, (II-1) cause the projector to project the specific content on the floor, and cause the camera to photograph the specific content projected on the floor. Obtain a content image, (II-2) have a deep learning network perform an operation on the content image, and have an image processing controller obtain information about the specific content obtained from the projector controller and the deep learning A process of determining whether the projector is broken, with reference to the judgment information output by the network,
In process (II) above,
If the processor determines that the autonomous vehicle is stopped, (II-3) before the specific content is projected on the floor by the projector, a pre-image before projection on the floor by the camera and Additional acquisition of a post-image after projection on the floor, (II-4) causing the image processing controller to calculate a difference image between the pre-image and the post-image, and calculating the difference image and the specific content. A projector failure diagnosis device characterized by further performing a process to determine whether the projector is broken by additional comparison. According to clause 17,
The processor causes the image processing controller to project information about the difference image onto a grid cell, compare the information projected on the grid cell with information about the specific content, detect a malfunction area, and thereby detect the malfunction area of the projector. A projector failure diagnosis device characterized in that it determines whether there is a failure. According to clause 11,
The specific content includes at least some of content indicating whether the autonomous vehicle is driving, content indicating actions to be performed by the autonomous vehicle within a preset time, and content indicating whether the autonomous vehicle is close to a pedestrian. A projector failure diagnosis device comprising corresponding text information. According to clause 11,
A projector failure diagnosis device, characterized in that the position of the projector is installed in front of the position of the camera, and the height of the projector is installed below the height of the camera.