KR102553130B1 - Image correction method for remote control of driving device - Google Patents

Abstract

In one embodiment of the present invention for solving the above problems, an image correction method for remote control of a mobile device is disclosed. The method may include acquiring driving image information of a mobile device and delay time information corresponding to the driving image information, correcting the driving image information based on the delay time information, and correcting the corrected driving image. and providing information to a user terminal, wherein the user terminal is a terminal related to a user who controls an operation of the mobile device based on the driving image information or the corrected driving image information. .

Description

Image correction method for remote control of mobile device {IMAGE CORRECTION METHOD FOR REMOTE CONTROL OF DRIVING DEVICE}

Various embodiments of the present invention relate to an autonomous driving system and a control method thereof, and more particularly, to a method for correcting driving image information referred to for remotely controlling a mobile device.

Today, the demand for autonomous driving of various vehicles (including vehicles or various means of transportation) is rapidly increasing. Autonomous driving may include remote driving or remote driving technology as one of its steps.

Remote driving refers to controlling the operation of a driving vehicle from a remote location outside the vehicle, rather than a user located inside the vehicle. For example, a user (eg, a controller) of a remote control center may control driving of a corresponding vehicle based on image information transmitted from a driving vehicle. In the case of such remote driving, since the user needs to perform driving-related control based on image information transmitted from the driving vehicle, the reliability of the transmitted image information may be very important.

On the other hand, in the case of a driving image, since the field of view is fixed mainly on the position of the camera module or the situation, there is a concern that a user located outside may not sufficiently recognize information around the vehicle in the field. In addition, a delay (eg, delivery delay) may occur in the process of receiving image information obtained from a camera module of a vehicle at a remote control center, and stability may not be secured in operation control of the vehicle due to the occurrence of such delay time. . Specifically, when a delay occurs in relation to a driving image, as the user controls driving or driving of the vehicle by referring to the delayed image information, coping with the situation may be delayed. This makes it difficult to quickly respond to a rapidly changing situation due to inaccurate image information, which may cause a major accident.

Accordingly, in the art, there is a need for a computing device that provides reliable driving image information through image correction corresponding to the occurrence of a delay while having improved readability so that a user located in a remote location can more accurately recognize the surroundings of the vehicle. Demand may exist.

Republic of Korea Patent Publication No. 10-2021-0055116 (2021.05.07)

An object of the present invention is to provide more reliable image information related to real-time driving by correcting a driving image in response to a delay time.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

In order to solve the above problems, an image correction method for remote control of a mobile device according to an embodiment of the present invention is disclosed. The method may include acquiring driving image information of a mobile device and delay time information corresponding to the driving image information, correcting the driving image information based on the delay time information, and correcting the corrected driving image. and providing information to a user terminal, wherein the user terminal is a terminal related to a user who controls an operation of the mobile device based on the driving image information or the corrected driving image information. .

In an alternative embodiment, the correcting of the driving image information may include differently determining a degree of correction of the driving image information according to a degree of delay corresponding to the delay time information.

In an alternative embodiment, the method may include receiving a motion control signal related to operation adjustment of the mobile device from the user terminal and correcting the driving image information based on the motion control signal and the delay time information. The operation control signal may include a control signal related to a movement direction of the mobile device.

In an alternative embodiment, the correcting the driving image information may include identifying an adjustment region in the driving image information based on the motion control signal and based on the identified adjustment region and the delay time information. and performing correction on the driving image information.

In an alternative embodiment, the driving image information may be acquired in plurality through a plurality of camera modules provided at various locations of the mobile device, and the user terminal may include among the plurality of driving image sub-information It may be characterized in that it includes a display unit for displaying at least a part.

In an alternative embodiment, the method may include classifying each of the plurality of driving image information into at least one of a main screen and a sub screen based on an operation control signal, and the corrected driving based on the main screen and the sub screen. The method may further include generating image information, wherein the main screen includes driving image information requiring correction in response to the occurrence of a time delay, and the sub screen may include driving image information not related to correction. .

In an alternative embodiment, the method may further include acquiring panoramic image information based on the plurality of pieces of driving image information and configuring the sub-screen based on the panoramic image information.

In an alternative embodiment, the method may further include determining to generate and transmit an emergency control signal to the mobile device based on the delay time information.

According to another embodiment of the present invention, an image calibration computing device for remote control of a mobile device is disclosed. The image correction computing device includes a memory for storing one or more instructions and a processor for executing the one or more instructions stored in the memory, the processor executing the one or more instructions, thereby enabling remote control of the aforementioned mobile device. It is possible to perform an image correction method for

According to another embodiment of the present invention, a computer program stored in a computer-readable recording medium is disclosed. The computer program may be combined with a computer, which is hardware, to perform an image correction method for remote control of a mobile device.

Other specific details of the invention are included in the detailed description and drawings.

According to various embodiments of the present invention, by correcting a driving image corresponding to a delay time, more reliable image information related to real-time driving may be provided.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a diagram schematically illustrating a system for performing an image correction method for remote control of a mobile device according to an embodiment of the present invention.
2 is an exemplary diagram illustrating a process of a user controlling an operation of a mobile device through a user terminal related to an embodiment of the present invention.
3 shows a block diagram of an image correction computing device for remote control of a mobile device according to an embodiment of the present invention.
4 shows an exemplary flow chart of an image correction method for remote control of a mobile device according to an embodiment of the present invention.
5 shows an exemplary flowchart of a method of correcting driving image information based on a motion control signal and driving image information related to another embodiment of the present invention.
6 is an exemplary diagram illustrating a process of correcting an image according to linear movement of a mobile device related to an embodiment of the present invention.
7 is an exemplary view illustrating a process of correcting an image according to rotational movement of a moving device according to another embodiment of the present invention.
8 is an exemplary diagram illustrating a main screen and a sub screen included in driving image information related to an embodiment of the present invention.
9 is an exemplary diagram for explaining that a main screen and a sub screen related to an embodiment of the present invention are variably included in corrected driving image information.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, only these embodiments are intended to complete the disclosure of the present invention, and are common in the art to which the present invention belongs. It is provided to fully inform the person skilled in the art of the scope of the invention, and the invention is only defined by the scope of the claims.

Terminology used herein is for describing the embodiments and is not intended to limit the present invention. In this specification, singular forms also include plural forms unless specifically stated otherwise in a phrase. As used herein, "comprises" and/or "comprising" does not exclude the presence or addition of one or more other elements other than the recited elements. Like reference numerals throughout the specification refer to like elements, and “and/or” includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various components, these components are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first element mentioned below may also be the second element within the technical spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings commonly understood by those skilled in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly specifically defined.

The term "unit" or "module" used in the specification means a hardware component such as software, FPGA or ASIC, and "unit" or "module" performs certain roles. However, "unit" or "module" is not meant to be limited to software or hardware. A “unit” or “module” may be configured to reside in an addressable storage medium and may be configured to reproduce one or more processors. Thus, as an example, a “unit” or “module” may refer to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. Functions provided within components and "units" or "modules" may be combined into smaller numbers of components and "units" or "modules" or may be combined into additional components and "units" or "modules". can be further separated.

The spatially relative terms "below", "beneath", "lower", "above", "upper", etc. It can be used to easily describe a component's correlation with other components. Spatially relative terms should be understood as including different orientations of elements in use or operation in addition to the orientations shown in the drawings. For example, if you flip a component that is shown in a drawing, a component described as "below" or "beneath" another component will be placed "above" the other component. can Thus, the exemplary term “below” may include directions of both below and above. Components may also be oriented in other orientations, and thus spatially relative terms may be interpreted according to orientation.

In this specification, a computer means any kind of hardware device including at least one processor, and may be understood as encompassing a software configuration operating in a corresponding hardware device according to an embodiment. For example, a computer may be understood as including a smartphone, a tablet PC, a desktop computer, a laptop computer, and user clients and applications running on each device, but is not limited thereto.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Although each step described in this specification is described as being performed by a computer, the subject of each step is not limited thereto, and at least a part of each step may be performed in different devices according to embodiments.

1 is a diagram schematically illustrating a system for performing an image correction method for remote control of a mobile device according to an embodiment of the present invention. As shown in FIG. 1, a system for performing an image correction method for remote control of a mobile device according to an embodiment of the present invention includes a mobile device 10, an image correction computing device 100, and a user terminal ( 20) and networks. Here, the system for performing the image correction method for remote control of the mobile device shown in FIG. 1 is according to an embodiment, and its components are not limited to the embodiment shown in FIG. 1, and as needed may be added, changed or deleted.

According to one embodiment of the present invention, the mobile device 10 may relate to a mobile body that generates power to perform driving or work. For example, the mobile device 10 may include a vehicle, a drone, an air vehicle, a motorcycle, a tank, and the like, as shown in FIG. 1 . However, it is not limited thereto, and may further include moving bodies (eg, excavators, forklifts, etc.) that drive by generating power or operate to increase productivity and stability in various industrial sites.

In embodiments, the mobile device 10 may be an autonomous driving capable device. A device capable of autonomous driving may refer to a device capable of unmanned driving through communication with the outside without a user inside the mobile device 10 . For example, a device capable of autonomous driving may determine a movement situation by itself, but may drive in relation to a command or a driving signal sent from the outside (eg, a user terminal). Such an autonomous driving device may include remote driving in which the operation of the driving vehicle is controlled from a remote location outside the vehicle, rather than a user located inside the vehicle.

According to an embodiment, the mobile device 10 may acquire image information in real time in relation to driving or motion, and transmit the obtained image information to at least one of the image correction computing device 100 and the user terminal 20 . Specifically, the mobile device 10 may include a plurality of camera modules, and may obtain driving image information related to driving through the camera modules provided. Here, driving image information may further include images related to work in addition to images related to driving. For example, when the moving object is related to an excavator, driving image information may be image information related to a process of performing an excavation work. The detailed description of the aforementioned mobile device and driving image information is only an example, and the present invention is not limited thereto.

That is, the driving image information obtained by the mobile device 10 may be image information that is used as a reference for the user terminal 20 to remotely control the operation of the corresponding mobile device 10 . A user of the user terminal 20 may recognize a surrounding situation of the mobile device 10 through driving image information, and may control an operation of the corresponding mobile device 10 in response thereto. For example, in response to the user terminal 20 receiving driving image information from the mobile device 10 indicating that the road on which the vehicle is traveling is related to a left curve road, the user refers to the corresponding driving image information and displays the information to the user terminal 20. It is possible to generate an operation control signal that controls the operation unit of the mobile device 10 to rotate in the left direction. The detailed description related to the aforementioned driving image information and motion control signal is only an example, and the present invention is not limited thereto.

In an embodiment, the mobile device 10 may be equipped with a plurality of camera modules. The plurality of camera modules may be used to acquire image information related to each area of the mobile device 10 . In other words, as the mobile device 10 is equipped with a plurality of camera modules, a plurality of driving image information on various fields of view may be acquired. This can provide a remotely located user with a broad field of view in relation to the work or driving environment. In an embodiment, the driving image information obtained from the mobile device 10 and transmitted to the user terminal 20 or the image calibration computing device 100 may include a plurality of driving image information corresponding to each of a plurality of camera modules. there is.

In various embodiments, the user terminal 20 may be connected to the image calibration computing device 100 or the mobile device 10 through a network, and may provide an operation control signal for moving the mobile device 10, Driving image information obtained as the mobile device 10 moves in response to the transmitted motion control signal is provided through the mobile device 10, or corrected driving image information related to correction of the corresponding driving image information is displayed as an image. It may be provided from the calibration computing device 100 .

Here, the network may refer to a connection structure capable of exchanging information between nodes such as a plurality of user terminals and devices. For example, the network includes a local area network (LAN), a wide area network (WAN), a world wide web (WWW), a wired and wireless data communication network, a telephone network, a wired and wireless television communication network, and the like.

In addition, here, the wireless data communication networks are 3G, 4G, 5G, 3GPP (3rd Generation Partnership Project), 5GPP (5th Generation Partnership Project), LTE (Long Term Evolution), WIMAX (World Interoperability for Microwave Access), Wi-Fi (Wi-Fi) Fi), Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), RF (Radio Frequency), Bluetooth network, A Near-Field Communication (NFC) network, a satellite broadcasting network, an analog broadcasting network, a Digital Multimedia Broadcasting (DMB) network, and the like are included, but are not limited thereto.

According to an embodiment, the user terminal 20 may include a display unit 21 for displaying driving image information. In an embodiment, a plurality of display units 21 may be provided. For example, driving image information may be obtained in plural pieces as each of the plurality of camera modules included in the mobile device 10 is obtained. In this case, a plurality of pieces of driving image information may be displayed on each of a plurality of display units of the user terminal 20 . For example, the mobile device 10 may be provided with three camera modules in relation to the front, left, and right directions, respectively, and thus, the user terminal 20 has three driving modes in relation to the front, left, and right directions, respectively. Image information may be received. In this case, the user terminal 20 may display driving image information on each of the three display units, as shown in FIG. 2 . For example, the user terminal 20 displays driving image information related to the front on a first display unit located in the center, and driving image information related to the left side is displayed on a second display unit located on the left side of the first display unit in the same way as in an actual driving environment. Displayed on the display unit, and driving image information related to the right side may be displayed on the third display unit located on the right side of the first display unit. Accordingly, the user 1 may be provided with an operating environment similar to an actual driving environment, that is, an operating environment similar to controlling an operation of the device inside the actual mobile device 10 .

Also, in the embodiment, the user terminal 20 may include a manipulation unit 22 that receives a user's input related to control of the mobile device 10 . For example, the manipulation unit 22 may be configured in the same way as the manipulation method of the actual mobile device 10 . For example, when the mobile device 10 is related to a vehicle, the control unit 22 may be composed of a steering wheel, a gear rod, an accelerator pedal, and a brake pedal. However, it is not limited thereto, and the control unit may further include various components related to actual driving or manipulation of the mobile device.

That is, the user 1 recognizes the driving or working environment of the mobile device 10 through the driving image screen displayed on the display unit 21, and applies an input to the control unit 22 in response to this, so that the mobile device 10 ) can be controlled.

In the embodiment, the user terminal 20 is a wireless communication device that ensures portability and mobility, and includes navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone) System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) terminal, smart All types of handheld-based wireless communication devices such as smartphones, smart pads, tablet PCs, etc. may be included, but are not limited thereto. For example, the user terminal may further include an artificial intelligence (AI) speaker and an artificial intelligence TV that provide various functions such as music appreciation and information search through interaction with a user based on a hot word.

According to an embodiment of the present invention, a system for performing an image correction method for remote control of a mobile device may include the image correction computing device 100 . The image correction computing device 100 may obtain driving image information from the mobile device 10 and may perform correction on the acquired driving image information. Specifically, the image correction computing device 100 may obtain driving image information and delay time information corresponding to the corresponding driving image information from the mobile device 10 . The delay time information is information related to a delay time generated in a process in which the mobile device 10 transmits image information in real time, and may be information about a time required for data to be transmitted or received through a network. For example, the delay time information may include a wireless communication link delay, transmission, and reception processing delay corresponding to transmission. This delay time information may be characterized in that it is changed according to the movement of the mobile device 10 .

For example, the delay time may be changed according to the separation distance between the user terminal 20 located in a remote location and the mobile device 10 . For another example, the delay time may be increased or decreased due to obstacles existing between the user terminal 20 located in a remote location and the mobile device 10 or interference of various radio signals. The specific descriptions related to the increase or decrease of the delay time described above are only examples, and the present invention is not limited thereto.

In an embodiment, the computing device 100 may correct driving image information based on the delay time information. Specifically, the driving image information acquired in real time from the mobile device 10 may include different delay time information according to the movement of the mobile device 10 . For example, at a first point in time when driving image information is obtained, delay time information indicating that a large delay occurs in a transmission process may be obtained, but at a second time point, delay time information indicating that a minimum delay occurs during a transmission process may be obtained. . This difference in delay time information may be caused by a change in communication speed corresponding to a change in the area caused by driving of the mobile device 10 . The specific description of the delay time information corresponding to each time point described above is only an example, and the present invention is not limited thereto.

That is, the real-time driving image information obtained from the mobile device 10 may have delay time information related to different degrees of delay for each time point according to the driving of the mobile device 10 . When a delay occurs in the driving image information, since the driving image information displayed on the user terminal 20 is the delayed driving image information, response to the situation may be delayed compared to the actual situation, and stability may not be secured. In particular, since the delay of the driving image information changes in real time according to the driving of the mobile device 10, the risk may be further maximized when the mobile device 10 is manipulated with reference to the driving image information related to the delay.

Accordingly, the computing device 100 of the present invention may correct driving image information based on delay time information that changes in real time according to driving of the mobile device. Correcting the driving image information based on the delay time information may mean correcting the driving image information to have a field of view similar to an actual driving environment even when a delay occurs. Such correction may mean, for example, adjusting the perspective of the field of view by enlarging or reducing a specific part in an image. In addition, the computing device 100 may provide driving image information with improved reliability to a user who manipulates or drives by correcting the driving image information through various correction methods according to the degree of delay.

Hereinafter, with reference to FIGS. 3 to 9 , a driving image correction method for remote control of a mobile device and specific effects thereof will be described in more detail below.

3 shows an exemplary block diagram of an image correction computing device 100 in accordance with one embodiment of the present invention.

As shown in FIG. 3 , the image correction computing device 100 may include a network unit 110 , a memory 120 and a processor 130 . The components included in the above image correction computing device 100 are examples, and the scope of the present invention is not limited to the above components. That is, additional components may be included or some of the above components may be omitted according to implementation aspects of the embodiments of the present invention.

According to an embodiment of the present invention, the image correction computing device 100 may include a network unit 110 that transmits and receives data to and from the mobile device 10 and the user terminal 20 . The network unit 110 may transmit and receive data for performing an image correction method for remote control of a mobile device according to an embodiment of the present invention to another computing device, a server, and the like. That is, the network unit 110 may provide a communication function between the image compensating computing device 100 and user terminals or between the image compensating computing device 100 and the mobile device 10 .

For example, the network unit 110 may receive driving image information from the mobile device 10 . Also, for example, the network unit 110 may transmit corrected driving image information to the user terminal 20 . Additionally, the network unit 110 calls a procedure to the image compensating computing device 100, thereby providing information between the image compensating computing device 100 and user terminals or between the image compensating computing device 100 and the mobile device 10. transfer can be allowed.

The network unit 110 according to an embodiment of the present invention includes a Public Switched Telephone Network (PSTN), x Digital Subscriber Line (xDSL), Rate Adaptive DSL (RADSL), Multi Rate DSL (MDSL), and VDSL ( Various wired communication systems such as Very High Speed DSL), Universal Asymmetric DSL (UADSL), High Bit Rate DSL (HDSL), and Local Area Network (LAN) may be used.

In addition, the network unit 110 presented in this specification includes Code Division Multi Access (CDMA), Time Division Multi Access (TDMA), Frequency Division Multi Access (FDMA), Orthogonal Frequency Division Multi Access (OFDMA), SC-FDMA ( Single Carrier-FDMA) and other systems.

In the present invention, the network unit 110 may be configured regardless of its communication mode, such as wired and wireless, and may be configured with various communication networks such as a personal area network (PAN) and a wide area network (WAN). can In addition, the network may be the known World Wide Web (WWW), or may use a wireless transmission technology used for short-range communication, such as Infrared Data Association (IrDA) or Bluetooth. The techniques described herein may be used in the networks mentioned above as well as other networks.

According to an embodiment of the present invention, the image correction computing device 100 may include a memory 120 . The memory 120 may store a plurality of application programs run in the image correction computing device 100 , data for correcting driving image data obtained from the mobile device 10 , and commands. At least some of these applications may be present in each image calibration computing device from the factory.

According to an embodiment, the memory 120 may store a computer program for performing an image correction method for remote control of a mobile device according to an embodiment of the present invention, and the stored computer program is executed by the processor 130. can be read and driven. In addition, the memory 120 may store any type of information generated or determined by the processor 130 and any type of information received by the network unit 110 . For example, the memory 120 may temporarily or permanently store input/output data. For example, the memory 120 may store driving image information and information about corrected driving image information corresponding to the corresponding driving image information. The detailed description of the information stored in the aforementioned memory is only an example, and the present invention is not limited thereto.

According to an embodiment of the present invention, the memory 120 is a flash memory type, a hard disk type, a multimedia card micro type, or a card type memory (eg, SD or XD memory, etc.), RAM (Random Access Memory, RAM), SRAM (Static Random Access Memory), ROM (Read-Only Memory, ROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory) -Only Memory), a magnetic memory, a magnetic disk, and an optical disk may include at least one type of storage medium.

According to one embodiment of the present invention, the image correction computing device 100 may include a processor 130 . The processor 130 may include one or more cores, and includes a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of the computing device. unit) may include a processor for data analysis.

According to one embodiment, the processor 130 may typically process overall operations of the image calibration computing device 100 . The processor 130 processes signals, data, information, etc. input or output through the components described above or runs an application program stored in the memory 120, thereby providing appropriate information to the mobile device 10 or the user terminal 20. Alternatively, it may provide or process a function. For example, the processor 130 may generate corrected driving image information corresponding to the driving image information.

For a specific example, the processor 130 may correct driving image information based on delay time information related to delay, and may determine to transmit the corrected driving image information to the user terminal 20 . In an embodiment, the corrected driving image information may be characterized in that a correction degree, a correction area, and the like are differently determined according to a delay amount and an operation control signal.

In addition, the processor 130 may obtain driving image information and delay time information corresponding to the corresponding driving image information from the mobile device 10, and may perform correction on the driving image information based on the delay time information. . Also, the processor 130 may determine to transmit the corrected driving image information to the user terminal 20 .

A method of correcting driving image information by the processor 130 will be described later in detail with reference to FIGS. 4 to 9 .

4 shows an exemplary flow chart of an image correction method for remote control of a mobile device according to an embodiment of the present invention. The order of steps for performing the image correction method for remote control of a mobile device shown in FIG. 4 may be changed as needed, and at least one or more steps may be omitted or added. That is, the following steps are only one embodiment of the present invention, and the scope of the present invention is not limited thereto.

According to an embodiment of the present invention, the processor 130 may obtain driving image information of the mobile device 10 and delay time information corresponding to the driving image information (S110). The driving image information may be reference image information for the user terminal 20 to control the operation of the corresponding mobile device 10 from a remote location. In the present invention, delay time information is information related to delay time occurring in the process of transmitting image information in real time, and may be information about time required for data to be transmitted or received through a network. For example, the delay time information may include information indicating that a delay of 10 ms has occurred corresponding to a first time point at which the driving image information is received. Specific numerical descriptions related to the aforementioned delay are only examples, and the present invention is not limited thereto.

According to an embodiment of the present invention, the processor 130 may correct driving image information based on the delay time information (S120). Specifically, the processor 130 may differently determine the degree of correction for the driving image information according to the degree of delay corresponding to the delay time information. Here, correction of the driving image information may mean correcting the driving image information so as to have a field of view similar to the actual driving environment even if a delay occurs. For example, correction of driving image information may include correction of enlargement and reduction, correction of rotation angle adjustment, correction of split screen adjustment, and the like.

According to an embodiment, the delay time information may be changed according to the movement of the mobile device 10 . For example, the delay time may be changed according to the separation distance between the user terminal 20 located in a remote location and the mobile device 10 . For another example, the delay time may be increased or decreased due to obstacles existing between the user terminal 20 located in a remote location and the mobile device 10 or interference of various radio signals. The specific descriptions related to the aforementioned delay time change are only examples, and the present invention is not limited thereto.

In an embodiment, the processor 130 may differently determine the degree of correction for the driving image information according to the degree of delay corresponding to the delay time information. In other words, the processor 130 may differently determine a correction degree for driving image information according to a change in delay time information. For example, when the delay time information includes information that a large delay occurs (eg, information that a delay occurs of 300 ms), the processor 130 may determine a correction degree for the driving image information to be large. For another example, when the delay time information includes information that almost no delay occurs (eg, information that the delay occurs by 1 ms), the processor 130 may determine a small correction degree for the driving image information. In other words, the processor 130 may determine a large correction degree in a section where a large delay occurs according to the movement of the mobile device 10 and a small correction degree in a section where a small delay occurs. Specific numerical description of the aforementioned delay time information is only an example, and the present invention is not limited thereto.

That is, the processor 130 may perform more appropriate correction in response to the delay by differently determining the degree of correction for the driving image information according to the degree of delay.

Also, according to an embodiment, the processor 130 may perform correction on driving image information based on the amount of change in delay time information. For example, when the delay time is changed from 300 ms to 10 ms in relation to the driving of the mobile device 10 (ie, when the delay time is shortened), the processor 130 performs correction to reduce driving image information to make it more visible. It can give you a wide field of view. For another example, when the delay time is changed from 20 ms to 250 ms in relation to the driving of the mobile device 10 (ie, the delay time is increased), the processor 130 performs correction to enlarge the driving image information to make it more visible. You can have a close-up view. The specific numerical description of the aforementioned delay time and the specific description of the correction method are only examples, and the present invention is not limited thereto.

That is, the processor 130 may more appropriately correct the driving image information in response to a change in delay time. In this case, the correction related to the enlargement or reduction of the driving image information may be for the user of the user terminal 20 to change the perspective.

According to an embodiment, the processor 130 may receive an operation control signal from the user terminal 20 and may correct driving image information based on the corresponding operation control signal and delay time information. A method of correcting driving image information based on an operation control signal and delay time information will be described below with reference to FIG. 5 .

According to the embodiment, the processor 130 may receive an operation control signal related to operation adjustment of the mobile device 10 from the user terminal 20 (S210). The operation control signal may include a control signal related to the movement direction of the mobile device 10 . In various embodiments, the operation control signal may further include control signals for specific tasks as well as control signals related to the moving direction of the mobile device 10, that is, driving. For example, when the mobile device 10 is related to a forklift, the operation control signal may further include a control signal related to the lifting and lowering of the forklift.

According to an embodiment of the present invention, the processor 130 may identify an adjustment area from the driving image information based on the motion control signal (S220). In detail, the processor 130 may identify an adjustment area in the driving image information based on direction information included in the motion control signal. Here, the adjustment region may be information about a region to be corrected in the driving video. That is, identifying the adjustment region may mean identifying a portion in the driving image that is expected to require correction. In an embodiment, driving image information corrected based on the adjustment region may be acquired. For example, corrected driving image information may be acquired as regions other than the adjustment region are removed from the driving image information.

As a specific example, referring to FIGS. 6 and 7 , when the operation control signal is related to a control signal for moving the mobile device 10 straight as shown in FIG. 6 (a), the processor 130, As shown in (b) of FIG. 6 , in the driving image information 200 , the center area may be identified as the adjustment area 201 .

As another example, as shown in (a) of FIG. 7 , when the operation control signal is a control signal related to a right turn of the mobile device 10, the processor 130 performs a driving image as shown in (b) of FIG. In the information 200, the right area can be identified as the adjustment area 201. The above-described specific descriptions related to the operation control signal and the adjustment region are only examples, and the present invention is not limited thereto.

That is, the position of the adjustment area can be identified in relation to the driving direction of the mobile device 10 .

According to an embodiment, the processor 130 may determine the size of the adjustment area based on the delay time information. Specifically, the processor 130 may determine the size of the adjustment region in response to the size of the generated delay time. For example, the processor 130 may determine a small size of the adjustment region when a large delay time occurs, and may determine a large size of the adjustment region when a small delay time occurs. In an embodiment, when the delay time is minimized, the processor 130 may identify the entire size of the driving image information as the adjustment area.

That is, the processor 130 may determine a smaller adjustment region related to correction as the delay time increases, and may determine a larger adjustment region as the delay time decreases. For example, referring to FIG. 6 , when a large delay occurs in response to driving image information 200 related to a situation in which the mobile device 10 is going straight, the processor 130 adjusts the adjustment area to be corrected. The size of (201) can be determined small. Accordingly, since correction (eg, magnification) related to a small area is performed, it is possible to provide a field of view that feels closer to the front area. On the contrary, when a small delay occurs, the processor 130 determines the size of the adjustment area to be corrected to be large, so that correction (eg, enlargement) related to a relatively large area is performed, corresponding to the front area. A field of view with a relatively reduced sense of proximity may be provided. In other words, the processor 130 may adjust the perspective of the driving image information by differently determining the size of the adjustment area according to the degree of occurrence of the delay time.

According to an embodiment of the present invention, the processor 130 may perform correction on driving image information based on the identified adjustment area and delay time information (S230).

Specifically, the processor 130 may remove an area other than the adjustment area from the driving image information, and may correct the driving image information based on the degree of delay time. Here, correction of the driving image information may mean correcting the driving image information so as to have a field of view similar to the actual driving environment even if a delay occurs.

As a specific example, referring to FIGS. 6 and 7 , an adjustment area is identified as shown in FIG. 6(b) based on an operation control signal related to straight movement of the mobile device 10, and a large delay time occurs at that time. In this case, the processor 130 may obtain the corrected driving image information 200a as shown in FIG. 6(c) through correction based on the corresponding adjustment area 201 . The corrected driving image information 200a may be acquired through magnification correction of the adjustment area 201 . In this case, the degree of expansion of the adjustment area 201 may be determined corresponding to the degree of delay time. For example, as the delay time increases, the magnification of magnification in relation to the adjustment area 201 may increase. In other words, as the delay increases, the adjustment area 201 is enlarged at a larger rate, thereby providing a view as if it is located closer to the front area.

For another example, when an adjustment area is identified as shown in (b) of FIG. 7 based on an operation control signal related to a right turn of the mobile device 10, and a large delay occurs at that time, the processor 130 Corrected driving image information 200a as shown in FIG. 7(c) may be acquired through correction based on the corresponding adjustment area 201 . The corrected driving image information 200a may be acquired through magnification correction of the adjustment area 201 . In this case, the degree of expansion of the adjustment area 201 may be determined corresponding to the degree of delay time. For example, as the delay time increases, the magnification of magnification in relation to the adjustment area 201 may increase. In other words, as the delay increases, the adjustment area 201 is enlarged at a larger rate, thereby providing a sense of view as if it is located closer to the area related to the right turn. The specific description related to the adjustment region described above is only an example, and the present invention is not limited thereto.

That is, by correcting the driving image information, which is different from the surrounding situation of the actual mobile device 10 due to the occurrence of delay, to be similar to the actual environment of the mobile device 10, the user receives more accurate cognitive information related to the manipulation of the mobile device 10. can provide.

In various embodiments, a plurality of driving image information may be obtained corresponding to the same viewpoint. Specifically, the mobile device 10 may be provided to include a plurality of camera modules, and a plurality of driving image information related to various views may be obtained through the provided camera modules. The processor 130 may receive a plurality of pieces of driving image information from the mobile device 10 . For example, the mobile device 10 may include three camera modules in relation to the front, left, and right sides respectively, and thus, the processor 130 may generate three driving images in relation to the front side, left side, and right side respectively. information can be received.

In an embodiment, when a plurality of driving image information is obtained, the processor 130 may determine which driving image information is to be used for correction. Determining whether to use the driving image information for correction may be performed based on an operation control signal generated by the user terminal 20 .

Specifically, the processor 130 may classify each of a plurality of pieces of driving image information into at least one of a main screen and a sub screen based on an operation control signal. Also, the processor 130 may generate corrected driving image information based on the main screen and the sub screen. Here, the main screen may include driving image information requiring correction in response to the occurrence of a time delay among a plurality of pieces of driving image information, and the sub screen may include driving image information not related to correction.

For example, referring to FIG. 8 , when the motion control signal is related to going straight, the processor 130 classifies the first driving image information related to the front area as the main screen 210 and relates to the left and right side respectively. Corrected driving image information may be obtained by classifying the second driving image information and the third driving image information into the sub screen 220 .

For another example, when the motion control signal is related to a left turn, the processor 130 classifies the first driving image information and the second driving image information related to the front and left directions, respectively, as the main screen, and the third driving image related to the right direction. Information can be classified into sub-screens.

In an additional embodiment, the driving image information may be related to panoramic image information. Panoramic image information may be image information that simultaneously captures a view angle higher than a general viewing angle through a plurality of cameras, and secures a wider viewing angle (eg, a 360-degree field of view) through a combination of the captured images. For example, the panoramic video image may be a video image related to a synthesized circular view viewed from above. The processor 130 may receive video images obtained through a plurality of cameras from the mobile device 10 and generate panoramic image information based on the received video images. In various embodiments, as shown in FIG. 9 , the processor 130 configures the main screen 210 through driving image information related to the driving direction of the mobile device and displays the sub screen 220 through panoramic image information. can be configured. In this case, since information on a wider field of view in relation to the driving environment can be provided, the convenience of controlling the mobile device 10 can be improved. The detailed description of the above-described main screen and sub screen is only an example, and the present invention is not limited thereto.

That is, the processor 130 may configure the main screen by identifying the driving direction of the mobile device 10 based on the motion control signal and correcting driving image information related to the driving direction based on the delay time. Also, the processor 130 may configure a sub-screen capable of assisting the main screen through driving image information not utilized for correction. In other words, by composing a main screen based on driving image information related to manipulation of the mobile device 10 among a plurality of driving image information and configuring a sub screen based on the remaining driving image information, the surroundings of the mobile device 10 are configured. It is possible to provide driving image information with improved readability with respect to the environment.

In a further embodiment, processor 130 may determine to generate and transmit an emergency control signal to mobile device 10 based on the latency information. Specifically, the processor 130 generates an emergency control signal when the generated delay time is equal to or greater than a preset threshold, and transmits the generated emergency control signal to the mobile device 10 so that the operation of the mobile device 10 is under emergency control. can be controlled as much as possible. For example, when an excessively large delay occurs, correction similar to an actual driving environment may not be performed even though driving image information is corrected. For example, as the delay time is very large, the adjustment area may be set to be very small, and a large magnification may be performed corresponding to the adjustment area. In this case, readability of the corrected driving image information may be remarkably deteriorated due to a high ratio of enlargement due to a large delay, and accordingly, identification of the driving environment may be difficult. In other words, when it is determined that too much delay occurs, the processor 130 determines that it is not easy to identify image information for controlling the mobile device 10 and transmits an emergency control signal to the mobile device 10. there is. Accordingly, the mobile device 10 can perform a preset control operation through the emergency control signal. Here, the preset control operation may include, for example, at least one of a control operation of gradually stopping the operation of the mobile device 10 and a control operation of moving the mobile device 10 to a position where the delay time is minimized. That is, when the delay is too great, the mobile device 10 may perform an emergency control, thereby preventing an accident due to a user's late manipulation based on the delayed driving image information.

According to an embodiment of the present invention, the processor 130 may provide the corrected driving image information to the user terminal (S130). The user terminal 20 may include a display unit 21 for displaying at least some of the plurality of driving image sub information.

In an embodiment, the display unit 21 of the user terminal 20 may be provided in plurality. For example, driving image information may be obtained in plural pieces as each of the plurality of camera modules included in the mobile device 10 is obtained. In this case, a plurality of pieces of driving image information may be displayed on each of a plurality of display units of the user terminal 20 . For example, the mobile device 10 may be provided with three camera modules in relation to the front, left, and right directions, respectively, and thus, the user terminal 20 has three driving modes in relation to the front, left, and right directions, respectively. Image information may be received. In this case, the user terminal 20 may display driving image information on each of the three display units, as shown in FIG. 2 . For example, the user terminal 20 displays driving image information related to the front on a first display unit located in the center, and driving image information related to the left side is displayed on a second display unit located on the left side of the first display unit in the same way as in an actual driving environment. Displayed on the display unit, and driving image information related to the right side may be displayed on the third display unit located on the right side of the first display unit. Accordingly, the user may be provided with a manipulation environment similar to an actual driving environment, that is, a manipulation environment similar to controlling an operation of the device inside the actual mobile device 10 .

In addition, since the driving image information that the user terminal 20 receives from the processor 130 and displays through the display unit 21 is driving image information corrected in response to the changing delay time information, it is more relevant to real-time driving. You can have improved reliability. That is, since driving image information corrected to be similar to the surrounding environment of the actual mobile device 10 is provided even in various delay time environments, it may have an effect of improving stability in remote control of the mobile device 10. there is.

Steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, implemented in a software module executed by hardware, or implemented by a combination thereof. A software module may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any form of computer readable recording medium well known in the art to which the present invention pertains.

Components of the present invention may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium. Components of the present invention may be implemented as software programming or software elements, and similarly, embodiments may include various algorithms implemented as data structures, processes, routines, or combinations of other programming constructs, such as C, C++ , Java (Java), can be implemented in a programming or scripting language such as assembler (assembler). Functional aspects may be implemented in an algorithm running on one or more processors.

Those skilled in the art will understand that the various illustrative logical blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein are electronic hardware, (for convenience) , may be implemented by various forms of program or design code (referred to herein as “software”) or a combination of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and the design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

Various embodiments presented herein may be implemented as a method, apparatus, or article of manufacture using standard programming and/or engineering techniques. The term "article of manufacture" includes a computer program, carrier, or media accessible from any computer-readable device. For example, computer-readable media include magnetic storage devices (eg, hard disks, floppy disks, magnetic strips, etc.), optical disks (eg, CDs, DVDs, etc.), smart cards, and flash memory. device (eg, EEPROM, card, stick, key drive, etc.), but is not limited thereto. Additionally, various storage media presented herein include one or more devices and/or other machine-readable media for storing information. The term “machine-readable medium” includes, but is not limited to, wireless channels and various other media that can store, hold, and/or convey instruction(s) and/or data.

It is to be understood that the specific order or hierarchy of steps in the processes presented is an example of example approaches. Based upon design priorities, it is to be understood that the specific order or hierarchy of steps in the processes may be rearranged while remaining within the scope of the present invention. The accompanying method claims present elements of the various steps in a sample order, but are not meant to be limited to the specific order or hierarchy presented.

The description of the presented embodiments is provided to enable any person skilled in the art to use or practice the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present invention. Thus, the present invention is not to be limited to the embodiments presented herein, but is to be construed in the widest scope consistent with the principles and novel features presented herein.

10: mobile device
20: user terminal
100: image correction computing device

Claims (10)

A method performed on one or more processors of a computing device, comprising:
obtaining driving image information of a mobile device and delay time information corresponding to the driving image information;
correcting the driving image information based on the delay time information; and
providing the corrected driving image information to a user terminal;
Including,
The user terminal,
Characterized in that it is a terminal related to a user who controls an operation of the mobile device based on the driving image information or the corrected driving image information,
In the step of correcting the driving image information,
receiving an operation control signal related to operation adjustment of the mobile device from the user terminal;
identifying an adjustment area in the driving image information based on the operation control signal; and
correcting the driving image information based on the identified adjustment area and the delay time information;
Including,
The operation control signal,
Including a control signal related to the direction of movement of the mobile device,
An image correction method for remote control of a mobile device.
According to claim 1,
In the step of correcting the driving image information,
Characterized in that the degree of correction for the driving image information is differently determined according to the degree of delay corresponding to the delay time information,
An image correction method for remote control of a mobile device.
delete delete According to claim 1,
The driving image information,
Characterized in that a plurality of cameras are obtained through a plurality of camera modules provided at various positions of the mobile device,
The user terminal,
Characterized in that it comprises a display unit for displaying at least some of the plurality of driving image sub information.
An image correction method for remote control of a mobile device.
According to claim 5,
The method,
classifying each of the plurality of pieces of driving image information into at least one of a main screen and a sub screen based on an operation control signal; and
generating the corrected driving image information based on the main screen and the sub screen;
Including more,
The main screen includes driving image information that needs to be corrected in response to the occurrence of time delay,
The sub screen includes driving image information not related to correction,
An image correction method for remote control of a mobile device.
According to claim 6,
The method,
obtaining panoramic image information based on the plurality of driving image information; and
configuring the sub-screen based on the panoramic image information;
Including more,
An image correction method for remote control of a mobile device.
According to claim 1,
The above method
determining to generate and transmit an emergency control signal to the mobile device based on the delay time information;
Including more,
An image correction method for remote control of a mobile device.
a memory that stores one or more instructions; and
a processor to execute the one or more instructions stored in the memory;
By executing the one or more instructions, the processor:
An apparatus that performs the method of claim 1 .
A computer program stored in a computer-readable recording medium to be combined with a computer, which is hardware, to perform the method of claim 1.

Images