KR102290026B1 - Waypoint pose detection and positioning method and apparatus using waypoints - Google Patents

Abstract

A method and apparatus for detecting and positioning a waypoint pose using a waypoint are disclosed. A driving control method using a waypoint includes: (a) acquiring a waypoint image after adjusting some of the setting values of an image sensor mounted on a vehicle; (b) detecting a waypoint pose from the waypoint image; and (c) generating an indoor route using the detected waypoint pose.

Description

Waypoint pose detection and positioning method and apparatus using waypoints

The present invention relates to a method and apparatus for detecting and positioning a waypoint pose using an indoor lighting type waypoint.

Recently, vehicles are equipped with lidar and image sensors, and various studies for autonomous driving are being conducted. Recently, a technology for recognizing a lane of a driving route using an image sensor mounted on the vehicle and displaying it in a cubic curve has been installed in an autonomous vehicle. However, the conventional method of detecting a surrounding object with an image sensor mounted on a vehicle, such as recognizing a lane on a driving route, requires a lot of time for image processing and has inaccuracies.

In addition, the prior art has the disadvantage that the process of obtaining the position of the red object on the image sensor coordinate system takes a lot of time, such as converting an image to a top-view, finding a target, and finding a location.

An object of the present invention is to provide a method and apparatus for detecting and positioning a waypoint pose using an indoor lighting type waypoint.

In addition, the present invention provides a method for detecting and positioning a waypoint pose using a waypoint capable of quickly and accurately extracting an indoor lighting-type waypoint without a complicated image processing process and using it to control driving to enable autonomous driving in an indoor space, and a method thereof to provide the device.

Another object of the present invention is to provide a method and apparatus for detecting and positioning a waypoint pose using a waypoint that is not affected by a surrounding environment due to low sensitivity to surrounding obstacles.

According to one aspect of the present invention, there is provided a waypoint pose detection and positioning method using an indoor lighting type waypoint.

According to an embodiment of the present invention, the method comprising the steps of: (a) acquiring a waypoint image after adjusting some of the setting values of an image sensor mounted on a vehicle; (b) detecting a waypoint pose from the waypoint image; and (c) generating an indoor route using the detected waypoint pose.

In step (a), the waypoint image may be captured after adjusting the aperture and shutter speed among the set values of the image sensor so that only the indoor lighting waypoint is included in the image captured by the image sensor.

In the step (b), the waypoint pose may be detected using each pixel value of each waypoint area in the waypoint image.

The detecting of the waypoint pose may include: deriving a waypoint pose on an image coordinate system using an average or a weighted sum of pixel values of each waypoint area; converting the waypoint pose on the image coordinate system into a waypoint pose on the image sensor coordinate system; and converting the waypoint pose on the sensor coordinate system into a waypoint pose on the vehicle coordinate system.

According to another aspect of the present invention, an apparatus for detecting and positioning a waypoint pose using an indoor lighting type waypoint is provided.

According to an embodiment of the present invention, there is provided a driving control device interlocked with a vehicle, comprising: a processor for controlling a portion of a setting value of an image sensor mounted on the vehicle to be adjusted when entering a room; an image acquisition unit for acquiring a waypoint image through the image sensor; and a detector configured to detect a waypoint pose from the waypoint image, wherein an indoor route is generated using the waypoint pose.

The processor may adjust the aperture and shutter speed among the set values of the image sensor so that only the indoor lighting waypoint is included in the image captured by the image sensor.

The detector may detect the waypoint pose by using each pixel value of each waypoint area in the waypoint image.

The detection unit derives a waypoint pose on an image coordinate system using an average or a weighted sum of pixel values of each waypoint region, and converts the waypoint pose on the image coordinate system into a waypoint pose on the image sensor coordinate system. It can be converted into a waypoint pose on the vehicle coordinate system by using the transformation relationship between the image sensor coordinate system and the vehicle coordinate system.

According to another embodiment of the present invention, when a vehicle enters a room, after adjusting some of the set values, the image sensor for taking a waypoint image; and a waypoint pose detection and positioning device for detecting a waypoint pose from the waypoint image and then generating an indoor route using the detected waypoint pose, wherein the vehicle is generated using the waypoint pose. A vehicle may be provided, characterized in that it is controlled to follow an indoor route.

By providing a method and apparatus for detecting and positioning a waypoint pose using a waypoint according to an embodiment of the present invention, there is an advantage in that it is possible to extract an indoor lit waypoint from a photographed image and then use it to control driving.

In addition, the present invention has an advantage in that it is possible to quickly and accurately extract an indoor lighting-type waypoint without a complicated image processing process and then use it to control driving so that autonomous driving in an indoor space is possible.

In addition, the present invention has an advantage that it is not affected by the surrounding environment due to low sensitivity to surrounding obstacles.

1 is a block diagram schematically illustrating an internal configuration of a waypoint pose detection and positioning device interlocked with a vehicle according to an embodiment of the present invention;
2 is a diagram illustrating an image taken before image sensor calibration.
3 is a view illustrating an image taken after image sensor calibration according to an embodiment of the present invention.
4 and 5 are diagrams for explaining a waypoint, a vehicle, and an image sensor coordinate system according to an embodiment of the present invention.
6 is a diagram illustrating a waypoint pose on an image coordinate system according to an embodiment of the present invention.
7 is a flowchart illustrating a method for detecting and positioning a waypoint pose using a waypoint according to an embodiment of the present invention.

As used herein, the singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as "consisting of" or "comprising" should not be construed as necessarily including all of the various components or various steps described in the specification, some of which components or some steps are It should be construed that it may not include, or may further include additional components or steps. In addition, terms such as "...unit" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software. .

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram schematically showing the internal configuration of a waypoint pose detection and positioning device interlocked with a vehicle according to an embodiment of the present invention, and FIG. 2 is a diagram illustrating an image taken before image sensor calibration, 3 is a view illustrating an image taken after image sensor calibration according to an embodiment of the present invention, and FIGS. 4 and 5 are a waypoint, vehicle and image sensor coordinate system according to an embodiment of the present invention. 6 is a diagram illustrating a waypoint pose on an image coordinate system according to an embodiment of the present invention. The waypoint pose detection and positioning apparatus 100 according to an embodiment of the present invention may be interlocked with a control unit (ECU) for controlling a vehicle or may be included as a part of the ECU.

Also, it is assumed that the vehicle is equipped with an image sensor. According to an embodiment of the present invention, the waypoint pose detection and positioning apparatus 100 detects a waypoint pose using an indoor lit waypoint photographed by an image sensor mounted on a vehicle, and uses this to detect an indoor route. make it possible to create

Hereinafter, even if there is no separate description, it should be understood that the waypoint pose detection and positioning device 100 is a device that interworks with a vehicle, and can obtain information from various sensors mounted on the vehicle.

Referring to FIG. 1 , a waypoint pose detection and positioning apparatus 100 according to an embodiment of the present invention includes an image acquisition unit 110 , a detection unit 115 , a driving control unit 120 , a memory 120 and a processor ( 125) is included.

The image acquisition unit 110 is a means for acquiring an image from an image sensor (not shown) mounted on the vehicle. The image sensor adjusts some of the set values when the vehicle enters the room under the control of the processor 125 . That is, the waypoint pose detection and positioning apparatus 100 may adjust the aperture and shutter speed among the set values of the image sensor at the moment the vehicle enters the room. At this time, the waypoint pose detection and positioning device 100 captures only the indoor lighting-type waypoint image in the image photographed through the image sensor, and sets the aperture and shutter speed of the image sensor so that other surrounding objects are not captured. can be adjusted

As described above, the image acquisition unit 110 may acquire an image (hereinafter, referred to as a waypoint image) captured to include only the waypoint by adjusting the set values of the aperture and shutter speed of the image sensor.

FIG. 2 is a diagram illustrating an image captured or previewed by an image sensor to which set values of aperture and shutter speed of the image sensor are not adjusted. As shown in FIG. 2 , it can be seen that, in the image captured before the setting values of the aperture and shutter speed of the image sensor are adjusted, in addition to the waypoint, other surrounding objects including the room are also photographed.

3 illustrates an image captured after adjusting the aperture and shutter speed of the image sensor according to an embodiment of the present invention. As shown in FIG. 3 , it can be seen that by adjusting the set values of the aperture and shutter speed of the image sensor, it can be seen that it is possible to obtain an image that does not include any surrounding objects other than the waypoint. Here, it is assumed that the image sensor is installed on the front upper part of the vehicle or the upper part of the vehicle.

The detection unit 115 is a means for detecting a waypoint pose in the waypoint image. This will be described in more detail.

The detector 115 acquires a waypoint positioning value on the image coordinate system from the waypoint image. For example, the detector 115 may derive each waypoint positioning value on the image coordinate system by using an average value of pixel values corresponding to the waypoint region in the waypoint image.

라 칭하기로 하며, 정규화된 RGB값을

와 같이 나타내기로 한다. 예를 들어, 각 픽셀값은 2ⁿ (n은 자연수)비트값을 가지는 것을 가정하기로 한다. 본 발명의 일 실시예에서는 n이 8인 것을 가정하기로 한다. 따라서, 각 픽셀값의 R,G,B값은 최대값이 255인 것을 가정하기로 한다. 각 픽셀의 최대값의 합(

)의 정규화된 값은 정규화된

와 같이 나타낼 수 있다. 영상좌표계상의 각 픽셀(i,j)의 정규화된 값은

로 나타내기로 한다. RGB values of each pixel of the waypoint image

It is called , and the normalized RGB value is

to be displayed as For example, it is assumed that each pixel value has a ^{bit value of 2 n} (n is a natural number). In an embodiment of the present invention, it is assumed that n is 8. Therefore, it is assumed that the maximum value of the R, G, and B values of each pixel value is 255. The sum of the maximum values of each pixel (

) is the normalized value of

can be expressed as The normalized value of each pixel (i, j) in the image coordinate system is

to be expressed as

로 나타내기로 한다. The coordinate value of each pixel on the image sensor is

to be expressed as

The detector 115 may detect pixels adjacent to each other having a normalized value greater than or equal to a reference value in the waypoint image as a waypoint area, and position the waypoint on the image coordinate system by averaging the coordinate values of pixels corresponding to the waypoint area. After obtaining the pixel value corresponding to , the positioning value on the image coordinate system can be obtained based on the image sensor calibration. If this is expressed as an equation, it is the same as in Equation 1.

는 각 성분에 대한 각 평균을 나타내는 함수이며,

는 웨이포인트 영역에 해당하는 픽셀들의 좌표값 성분을 나타낸다. here,

is a function representing each mean for each component,

denotes the coordinate value component of pixels corresponding to the waypoint area.

In an embodiment of the present invention, as described above, a waypoint positioning value on an image coordinate system may be derived by deriving an average value of a coordinate value component of pixel values for each waypoint region in a waypoint image.

As another example, the detector 115 may derive a waypoint positioning value on the image coordinate system by using a weighted sum. This will be described in more detail as follows.

The detector 115 obtains a pixel value corresponding to the waypoint region on the image coordinate system by weighted summing the coordinate values of the waypoint region in the waypoint image, and then uses the image sensor calibration to determine the position value on the final image coordinate system. can be derived

If this is expressed as an equation, it is the same as in Equation 2.

As described above, the waypoint positioning value on the image coordinate system derived using the pixel value corresponding to the waypoint area is shown in FIG. 6 .

When the positioning value on the image coordinate system is derived, the detector 115 may convert the waypoint positioning value on the image coordinate system into a positioning value on the image sensor coordinate system.

와 같이 나타낼 수 있다. 또한, 영상 센서 좌표계와 마찬가지로 차량 좌표계 또한 동일하게 나타낼 수 있으며, 이는

와 같이 나타낼 수 있다. A waypoint, a vehicle, and an image sensor coordinate system will be briefly described with reference to FIGS. 4 and 5 . As shown in FIG. 4 , in the image sensor coordinate system, the x-axis and y-axis are respectively determined based on the image sensor position, which

can be expressed as In addition, like the image sensor coordinate system, the vehicle coordinate system can also be represented in the same way, which

can be expressed as

)에 대해 설명하면, 차량에 장착된 영상 센서의 설치 위치를 영상 센서 좌표계의 원점으로 설정한 후 영상 센서의 원점과 각 웨이포인트까지의 각도를 이용하여

가 계산될 수 있다. 도 4 및 도 5에 도시된 바와 같은 각 웨이포인트, 차량 및 영상 센서의 구조적인 관계를 이용하여 영상좌표계상의 웨이포인트 측위값은 영상센서좌표계상의 웨이포인트 측위값으로 변환될 수 있다. As shown in FIG. 5 , the c-axis (

), set the installation position of the image sensor mounted on the vehicle as the origin of the image sensor coordinate system, and then use the angle from the origin of the image sensor to each waypoint.

can be calculated. A waypoint positioning value on the image coordinate system may be converted into a waypoint positioning value on the image sensor coordinate system using the structural relationship between each waypoint, vehicle, and image sensor as shown in FIGS. 4 and 5 .

The detector 115 may finally detect the waypoint pose by converting the waypoint positioning value on the image sensor coordinate system into a waypoint positioning value on the vehicle coordinate system. The detector 115 may convert a waypoint positioning value on the image coordinate system into a positioning value on the vehicle coordinate system by using a transformation relationship between the image sensor coordinate system and the vehicle coordinate system.

식으로 표시할 수 있다. As such, the waypoint positioning value on the vehicle coordinate system is calculated using curve fitting.

can be expressed as

The memory 120 is a means for storing instructions (program codes) for waypoint pose estimation and driving control using the same according to an embodiment of the present invention, various data derived from this process, and the like.

The processor 125 includes internal components (eg, the image acquisition unit 110 , the detection unit 115 , the memory 120 , etc.) of the waypoint pose detection and positioning apparatus 100 according to an embodiment of the present invention. ) is a means to control

Also, when the vehicle enters the room, the processor 125 may control to change the set values of the aperture and shutter speed of the image sensor mounted on the vehicle. In this case, as already described above, the processor 125 may control to change the setting values of the aperture and shutter speed of the image sensor so that only the indoor lit waypoint is included in the image captured (or previewed) by the image sensor. .

In addition, the processor 125 may generate a route using the waypoint pose detected by the detector 115 and control the vehicle to travel along the generated route. As a result, the waypoint pose detection and positioning apparatus 100 according to an embodiment of the present invention detects a waypoint pose from an indoor waypoint image and uses it to generate an indoor route so that the vehicle can autonomously drive indoors. There are advantages to being able to

7 is a flowchart illustrating a method for detecting and positioning a waypoint pose using a waypoint according to an embodiment of the present invention.

In step 710, the waypoint pose detection and positioning apparatus 100 determines whether the vehicle has entered the room.

If it does not enter the room, it waits in step 710 .

However, if it is determined that the vehicle has entered the room, the waypoint pose detection and positioning apparatus 100 adjusts at least some set values of the image sensor in step 715 . As already described above, the image sensor may be a camera mounted on a vehicle. The waypoint pose detection and positioning apparatus 100 may adjust the aperture and shutter speed of the image sensor so that only the indoor lit waypoint is included in the image photographed (or previewed) by the image sensor mounted on the vehicle. The process of adjusting the set value of the image sensor mounted on the vehicle may be performed only once immediately after the vehicle enters the room.

In operation 720, the waypoint pose detection and positioning apparatus 100 acquires a waypoint image.

In step 725, the waypoint pose detection and positioning apparatus 100 detects each waypoint pose from the waypoint image.

First, the waypoint pose detection and positioning apparatus 100 detects each waypoint area in a waypoint image, and then derives each waypoint positioning value on the image coordinate system. For example, the waypoint pose detection and positioning apparatus 100 may obtain each waypoint positioning value on the image coordinate system by using the average or weighted sum of pixel values of each waypoint area, as described above in FIG. 1 . .

Next, the waypoint pose detection and positioning apparatus 100 converts each waypoint positioning value on the image coordinate system into a waypoint positioning value on the image sensor coordinate system, and then converts it to a waypoint positioning value on the vehicle coordinate system to final waypoint pose can be detected. Since this is the same as that described in FIG. 1 , the overlapping description will be omitted.

In this way, the waypoint pose detected by the waypoint pose detection and positioning device 100 is used to generate an indoor route, and by controlling the vehicle to follow the generated indoor route, it is possible to enable autonomous driving of the vehicle indoors. there is.

That is, the waypoint pose detection and positioning apparatus 100 adjusts the setting values of the aperture and shutter speed of the image sensor so that the image sensor mounted on the vehicle shoots only the waypoint, then acquires the waypoint image, and then performs the waypoint pose. each can be detected. For this reason, the vehicle including the corresponding waypoint pose detection and positioning device 100 can generate an indoor route using the detected waypoint pose, and the vehicle can follow the indoor route to enable autonomous driving. there is.

The apparatus and method according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the computer readable medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the computer software field. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - Includes magneto-optical media and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

So far, the present invention has been looked at focusing on the embodiments thereof. Those of ordinary skill in the art to which the present invention pertains will understand that the present invention can be implemented in modified forms without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

100: driving control device
110: image acquisition unit
115: detection unit
120: memory
125: processor

Claims (10)

(a) When the vehicle enters the room, the image captured by the image sensor of the vehicle includes only the indoor lit waypoint, and the image mounted on the vehicle so that other surrounding objects are not included except for the indoor lit waypoint acquiring a waypoint image after adjusting the aperture and shutter speed among the set values of the sensor - the image sensor is installed in the front or upper part of the vehicle;
(b) detecting a waypoint pose from the waypoint image; and
(c) generating an indoor route using the detected waypoint pose,
Step (b) is,
A waypoint positioning value on the image coordinate system is derived using each pixel value of each waypoint area in the waypoint image, and the waypoint positioning value on the image coordinate system is calculated using the structural relationship between the vehicle and the image sensor. A waypoint pose detection and positioning method using a waypoint, characterized in that the waypoint pose is detected by converting it into a waypoint positioning value on an image sensor coordinate system.
delete delete According to claim 1,
The step of detecting the waypoint pose comprises:
deriving a waypoint pose on an image coordinate system using an average or a weighted sum of pixel values of each waypoint region;
converting the waypoint pose on the image coordinate system into a waypoint pose on the image sensor coordinate system; and
and converting the waypoint pose on the sensor coordinate system into a waypoint pose on the vehicle coordinate system.
A computer-readable recording medium recording program code for performing the method according to claim 1 .
In a waypoint pose detection and positioning device using a waypoint interlocked with a vehicle,
Among the set values of the image sensor mounted on the vehicle, so that only the indoor lit waypoint is included in the image captured by the image sensor of the vehicle when the vehicle enters the room, and other surrounding objects other than the indoor lit waypoint are not included a processor that controls the aperture and shutter speed to be adjusted;
an image acquisition unit for acquiring a waypoint image through the image sensor; and
A detection unit for detecting a waypoint pose from the waypoint image,
An indoor route is created using the waypoint pose,
The detection unit,
A waypoint positioning value on the image coordinate system is derived using each pixel value of each waypoint area in the waypoint image, and the waypoint positioning value on the image coordinate system is calculated using the structural relationship between the vehicle and the image sensor. A waypoint pose detection and positioning device, characterized in that the waypoint pose is detected by converting it into a waypoint positioning value on an image sensor coordinate system.
delete delete 7. The method of claim 6,
The detection unit,
The waypoint pose on the image coordinate system is derived using the average or weighted sum of the pixel values of each waypoint region, and the waypoint pose on the image coordinate system is converted into a waypoint pose on the image sensor coordinate system. A waypoint pose detection and positioning device, characterized in that it is converted into a waypoint pose on the vehicle coordinate system by using a transformation relationship on the vehicle coordinate system.
When the vehicle enters the room, a waypoint image is taken after adjusting the aperture and shutter speed among the set values so that only the indoor lit waypoint is included in the photographed image and no surrounding objects other than the indoor lit waypoint are included. sensor; and
a waypoint pose detection and positioning device for detecting a waypoint pose from the waypoint image and generating an indoor route using the detected waypoint pose,
The waypoint pose detection and positioning device comprises:
A waypoint positioning value on the image coordinate system is derived using each pixel value of each waypoint area in the waypoint image, and the waypoint positioning value on the image coordinate system is calculated using the structural relationship between the vehicle and the image sensor. Detects the waypoint pose by converting it into a waypoint positioning value on the image sensor coordinate system,
The vehicle is controlled to follow an indoor route created using the waypoint pose.

Images