JP2019071085A - Method and system for providing position or movement information for controlling at least one function of vehicle - Google Patents

Abstract

To provide a method for providing position information for controlling at least one function of a vehicle provided with a vehicle coordinate system, and a method and a system which provide movement information indicative of a movement associated with a mobile device relative to the vehicle coordinate system to control at least one function of the vehicle according to the movement.SOLUTION: The method includes the steps of: receiving image information associated with at least one image of at least one part of a vehicle captured by a camera of a mobile device, the mobile device being separate from the vehicle; determining a position associated with the mobile device in the vehicle coordinate system according to the image information associated with the at least one image; and providing position information indicative of the position associated with the mobile device in the vehicle coordinate system to control at least one function of the vehicle according to this position.SELECTED DRAWING: Figure 1

Description

  The present disclosure relates to methods and systems for providing position or motion information for controlling at least one function of a vehicle.

  Recently, modern vehicles such as automobiles have various systems or functions that can be adjusted to specific locations within the vehicle to have the desired performance. For example, the in-vehicle acoustic system can be adjusted to optimize the acoustic experience for the driver seated in the front seat and the passenger seated in the passenger seat and / or the rear seat. Systems such as vehicle ambient light systems and climate systems are similar. Users rarely do so as they may struggle to specify their position in 3D space to coordinate one or more functions of such a system. In particular, it is difficult or even impossible for the user to specify the desired position inside the motor vehicle in order to adjust or control such a system of the motor vehicle. Furthermore, changing from one desired position to another desired position inside the car is also difficult for the user.

  Zhang et al., In reference [1], propose a method for self-calibration of the camera position relative to the ground surface for a camera mounted on a vehicle. They solve the problem that the height of the mounted camera depends on the load of the vehicle which will change from time to time. Their method determines the height of the camera above the ground while the vehicle is moving. They first define the ground surface in camera coordinates according to two image features in at least two camera images, and then determine the height of the camera above the ground surface. Here, determining the height of the camera includes calculating an inner product between the average of the feature points and the normal to the ground surface.

  Breed, in reference [7], discloses a method for obtaining information about occupancy elements in a vehicle. This method requires an image of the area in the vehicle where the occupancy element may be located, and uses a trained neural network to classify the elements in the image.

  Breed et al., In reference [8], acquire images of occupants or occupants in a vehicle from a single camera located in the vehicle and analyze the images acquired from the single camera, It is proposed to determine the type of occupant or occupant.

  Grossmann et al., In [2], disclose determining the position of the camera relative to the display according to the image of the visual content displayed on the display captured by the camera. Osman, in reference [3], proposes a method for determining the position of the camera of the gaming machine relative to the display device. He discloses determining the position of the camera relative to the display by analyzing an image of the visual content displayed on the display captured by the camera. However, references [2, 3] do not motivate the display device to be mounted on a vehicle and to control the functions of the vehicle based thereon.

  An object of the present invention is to provide a method and system that allow a user to provide information suitable for controlling at least one function of a vehicle based on position or movement.

  According to a first aspect, a method for providing position information for controlling at least one function of a vehicle provided with a vehicle coordinate system, the method being captured by a camera of a mobile device Receiving image information associated with at least one image of at least one portion of a vehicle, the mobile device being separate from the vehicle, and a vehicle coordinate system according to the image information associated with the at least one image Determining a position associated with the mobile device within the network, and providing position information indicating a position associated with the mobile device within the vehicle coordinate system to control at least one function of the vehicle according to the position; Methods are provided.

  According to the invention, modern vehicles are often equipped, for example, with a display mounted towards the driver and / or passenger seated in the vehicle at a known position relative to the vehicle inside the vehicle. The fact of being equipped, as well as the fact that personal mobile devices such as cell phones or tablet computers are often equipped with cameras, can be used to advantage.

  According to the invention, at least one function of the vehicle may be controlled based on the determined position associated with the mobile device comprising the camera. This position is determined by capturing at least one image of a portion of the vehicle using a camera. For example, the captured portion of the vehicle includes visual content displayed on a display mounted on a car. Thus, the present invention provides methods and systems that allow the user to determine the position inside the vehicle to control at least one function of the vehicle based on the determined position. For example, the vehicle's sound system, lighting system and / or air conditioning functions may be controlled according to the position of the mobile device.

  In order to achieve this object, according to one embodiment, the present invention is displayed, for example, on a display device mounted on a vehicle, a camera of a mobile device and a display device captured by a camera of the mobile device It is proposed to determine the position of the mobile device within the coordinate system of the vehicle by using the image of the visual content. Furthermore, at least one function of the vehicle is controlled based on the determined position. For example, the functionality may relate to a subsystem of the vehicle, such as an acoustic system, an ambient light system, an air conditioning system, or a combination thereof.

  According to one embodiment, the method further comprises providing or receiving the position of at least one portion of the vehicle within the vehicle coordinate system.

  According to one embodiment, at least one portion of the vehicle includes visual content displayed on a display mounted on the vehicle.

  According to one embodiment, determining the position associated with the mobile device in the vehicle coordinate system comprises: at least one portion of the vehicle in capturing the at least one image according to image information associated with the at least one image Determining the camera position of the camera and determining the position associated with the mobile device in the vehicle coordinate system according to the camera position in the vehicle coordinate system and the position of at least one portion of the vehicle.

  According to a further embodiment, it is preferred that the at least one image is a first image and the camera is a first camera and determining the position associated with the mobile device in the vehicle coordinate system is the Receiving image information associated with the second image of the at least one portion of the vehicle captured by the two cameras, and at least one portion of the vehicle according to the image information associated with the first image and the second image And restore the model of.

  In a further development, the method comprises receiving image information associated with a third image captured by a third camera of the mobile device, the third image being at least a portion of the reconstructed model (At least a portion of the vehicle may differ from at least one portion of the vehicle described above, provided that at least a portion of the vehicle is included in the reconstructed model) Determining the camera position of the third camera with respect to the restored model in capturing the third image according to at least a portion of the restored model and the image information associated with the third image. And the mobile device in the vehicle coordinate system according to the camera position of the third camera Further includes determining a position.

  According to one embodiment, the method determines the camera position of the first camera relative to at least one portion of the vehicle in capturing the first image according to at least a portion of the image information associated with the first image And determining the camera position of the second camera with respect to at least one portion of the vehicle when capturing the second image according to at least a portion of the image information associated with the second image.

  According to a further development, at least one part of the vehicle is a first part of the vehicle and the method receives image information associated with a fourth image captured by a fourth camera of the mobile device A fourth image capturing at least a portion of the first portion of the vehicle and at least a second portion of the vehicle; a restored model, the first portion of the vehicle and the vehicle And D. expanding to include the second portion.

  Preferably, the reconstructed model at least describes depth information of at least a portion of the vehicle.

  According to one embodiment, at least two of the first camera, the second camera, the third camera, and the fourth camera are the same camera.

  According to another embodiment, at least two of the first camera, the second camera, the third camera, and the fourth camera are different cameras.

  According to one embodiment, the at least one function is a function controlled according to position associated with at least one of a vehicle acoustic system, a vehicle ambient light system and a vehicle air conditioning system.

  According to a first aspect, there is also provided a system for providing position information for controlling at least one function of a vehicle provided with a vehicle coordinate system, the system being captured by a camera of a mobile device A first processing device configured to receive image information associated with at least one image of at least one portion of the at least one vehicle, the mobile device being separate from the vehicle And a first processing unit configured to determine a position associated with the mobile device within the vehicle coordinate system according to the image information associated with the at least one image, and to control at least one function of the vehicle according to the position Configured to provide position information indicative of a position associated with the mobile device within the vehicle coordinate system Comprising a second processor, a system is provided.

  According to a second aspect, a system for providing movement information for controlling at least one function of a vehicle provided with a vehicle coordinate system, the system comprising at least one camera captured by at least one camera of a mobile device. A first processing unit configured to receive image information associated with two images, wherein the mobile device is separate from the vehicle, and each of the at least two images captures at least one portion of the vehicle A first processing unit configured to determine motion associated with the mobile device relative to the vehicle coordinate system according to image information associated with the at least two images; and at least one of the vehicle according to the motion Direct the motion associated with the mobile device to the vehicle coordinate system to control one function And a second processing unit configured to provide dynamic information, a system is provided.

  The following and other embodiments described herein are equally applicable for both the first and second aspects.

  According to one embodiment, the first processing unit and the second processing unit are the same processing unit.

  According to one embodiment, at least one of the first processing unit and the second processing unit is comprised in the mobile device. According to another embodiment, at least one of the first processing device and the second processing device is included in the vehicle. According to a further embodiment, at least one of the first processing unit and the second processing unit is comprised in a computing device in communication with the mobile device and the vehicle.

  According to a second aspect, there is also provided a method for providing motion information for controlling at least one function of a vehicle provided with a vehicle coordinate system, the method comprising the steps of: Receiving image information associated with the at least two images captured by the camera, the mobile device being separate from the vehicle, each of the at least two images capturing at least one portion of the vehicle And determining motion associated with the mobile device relative to the vehicle coordinate system according to image information associated with the at least two images, and associated with the mobile device relative to the vehicle coordinate system to control at least one function of the vehicle according to the motion Providing the exercise information for instructing the exercise; .

  According to one embodiment, the movement associated with the mobile device relative to the vehicle coordinate system may be determined according to two positions associated with the mobile device within the vehicle coordinate system. The two positions associated with the mobile device can each be determined according to the image information associated with the at least two images. At least one portion of the vehicle captured within each of the at least two images may have a common portion or may be different.

  According to a further embodiment, the movement associated with the mobile device with respect to the vehicle coordinate system can be determined according to two camera positions of the at least one camera in capturing the at least two images.

  The aspects and embodiments of the invention described above or below with respect to the first aspect of determining the position associated with the mobile device also apply to the second aspect with respect to determining the movement associated with the mobile device. It is equally applicable.

  According to one embodiment, the movement associated with the mobile device relative to the vehicle coordinate system may be determined according to the image position of at least one portion of the vehicle captured within each of the at least two images. This requires that at least one portion of the vehicle captured within each of the at least two images be the same or have a common portion. However, this does not require the known position of at least one part of the vehicle in the vehicle coordinate system.

  According to the invention, at least one function of the vehicle may be controlled based on the determined movement associated with the mobile device. For example, the sound system of the vehicle, the lighting system and / or the function of air conditioning may be controlled according to the movement of the mobile device. Motion may be used to change the volume of the sound system. The illumination system may change the current illumination position to a new illumination position according to the determined movement of the mobile device. For example, the determined movement may define how far away from the current irradiation position to the new irradiation position. In another example, the determined movement may define a rotation, which allows the illumination system to rotate from the current illumination direction to the new illumination direction according to this rotation.

  Preferably, the mobile device is a handheld device, such as a mobile phone, a tablet computer or a mobile computer.

  For example, at least one portion of the vehicle has a position in the vehicle coordinate system that may be provided by the manufacturer of the vehicle.

  At least one portion of the vehicle captured in the image could be any physical portion of the vehicle. At least one portion of the vehicle may include a display mounted on the vehicle, or a portion of the display. At least one portion of the vehicle may further include one or more visual content displayed on the display device.

  The display device is a device for visual presentation of information. The display may be based on any display technology or display material, such as a cathode ray tube (CRT), a light-emitting diode display (LED) and a liquid crystal display (LCD). It will be possible. The display device may include a two-dimensional flat display or a display having a curved shape. The display may also be a foldable display including a plurality of planar sub-displays, each of which could be moved with the other.

  The display device may be mounted on a vehicle. It is preferably mounted towards the eyes of the driver and / or the passenger seated in the vehicle. Thereby, the driver and / or the passenger will be able to view the visual content displayed on the display device. For example, the display may be mounted on the front control panel of the vehicle or mounted on the back of the front seat of the vehicle.

  The display could be used to display menus, maps, graphical user interfaces of software programs, etc.

  Visual content is any information that an anatomical eye or optical imaging device can visually perceive. For example, visual content may emit or reflect visible light that may be captured by the human eye or camera. Visual content may also emit or reflect invisible light that may not be captured by the human eye but may be captured by a camera. The visual content could be text, diagrams, computer-generated or camera-captured images, symbols, pictures, combinations thereof, or portions of each of them. For example, the visual content may be menus, buttons, icons, digital maps, graphical user interfaces of software programs, combinations thereof, or portions of each of them. Visual content can be displayed on a display device.

  The mobile device is portable and comprises one or more cameras. One or more cameras may have a known position relative to the mobile device. The mobile device could be a camera only. For example, the mobile device may be a camera, a first camera, a second camera, a third camera, and / or a fourth camera. That is, the mobile device and each camera are the same device.

  In one embodiment, the mobile device may have at least one processing unit, such as a computing processor. In another embodiment, the mobile device may not have a computing processor, but a transmitter for transmitting data (e.g., image data of an image captured by a camera) to another device (e.g., a vehicle) May be included. The mobile device could be, but is not limited to, a mobile phone, a tablet computer, or a laptop. The camera may be separate from the mobile device, but communicates with the mobile device via a cable or wirelessly.

  One or more cameras are optical imaging devices that could capture image information of optical information.

  The proposed invention can be applied using any camera that provides an image. It is not limited to cameras that provide color images in RGB format. It can also be applied to cameras that provide monochrome images, such as grayscale images, as well as any other color format. The camera may further provide an image having depth data. Depth data need not be provided at the same resolution as the (color / grayscale) image. Cameras that provide images with depth data are often referred to as RGB-D cameras. The RGB-D camera system could be a time of flight (TOF) camera system. The camera may also capture light that is invisible to the human eye, such as infrared light. For example, the camera may be a thermal imaging camera.

  The position of the camera in the vehicle coordinate system may include translation, rotation, or a combination thereof in the vehicle coordinate system.

  The visual appearance of the visual content displayed describes the shape, texture, geometry or a combination thereof. The visual appearance may or may not include color information.

  The physical geometry of an object describes the size, shape, dimensions, planarity, or combinations thereof of the object when it is in the real world.

  The resolution of a display is the number of distinct pixels in each dimension that can be displayed on its display area. The display may have a known resolution. The display device may further have a known physical geometry of its display area. The physical geometry of the display refers to the physical geometry of the display area of the display. By having the pixel position of the displayed visual content in the coordinate system of the display, and the resolution and physical geometry of the display, the spatial relationship between the displayed visual content and the display can be determined. This defines the position of the displayed visual content relative to the display device. Additionally, the physical geometry of the displayed visual content can also be determined. The position of the displayed visual content in the vehicle coordinate system may be determined according to the position of the displayed visual content relative to the display and the position of the display relative to the vehicle coordinate system.

  The camera position of the camera relative to at least a portion of the vehicle while the first image is captured by the first camera may be determined according to at least a portion of the image. Having at least a portion of the physical geometry of the vehicle, various vision-based camera position estimation methods, such as methods based on 2D-3D point correspondence (see, for example, reference [9]) for determining camera position It can be used.

  In particular, it is preferred to have at least a portion of the vehicle containing the displayed visual content. Therefore, the camera image captures the displayed visual content. Visual content could provide vivid textures and features (eg, contrasting corners or edges) that enhance vision-based camera position estimation. In this example, the camera position of the camera can be determined with respect to the display device mounted on the vehicle. The position of the display relative to the vehicle coordinate system could, for example, be provided by the manufacturer of the vehicle.

  Having the known visual appearance of the displayed visual content and its physical geometry determine the camera position of the camera relative to the visual content using an accurate scale factor according to at least a portion of the image information of the first image can do. The estimation of the camera position could be based on the correspondence between image features of the captured image and corresponding features of the visual content displayed on the display. At this time, the camera position in the vehicle coordinate system is determined according to the camera position with respect to the displayed visual content and the position of the displayed visual content in the vehicle coordinate system.

  The camera position of the camera within the vehicle coordinate system while the camera captures an image can be determined from the camera position relative to at least one portion of the vehicle and the position of at least one portion of the vehicle within the vehicle coordinate system.

  In one example, the position of the camera within the vehicle coordinate system could determine the position that can be provided to the vehicle to control at least one function of the vehicle. The position associated with the mobile device may be the same as the position of the camera in the vehicle coordinate system. The position associated with the mobile device may have a spatial displacement from the position of the camera within the vehicle coordinate system.

  In another example, the position of the camera within the vehicle coordinate system may not be appropriate to specify the position associated with the mobile device. For example, the position of the camera may not be the desired position to be provided to the vehicle to control at least one function.

  According to an embodiment, the invention proposes to further capture a second image of at least one part of the vehicle by a second camera of the mobile device (the above-mentioned camera and image are respectively the first camera And the first image). The second image and the first image are captured when the second camera and the first camera are in different positions.

  The second camera position of the camera relative to at least one portion of the vehicle while the second image is captured may be determined according to at least a portion of the second image. The disclosed method of determining the camera position of the first camera may be used to determine the camera position of the second camera.

  A model of at least one portion of the vehicle may be restored according to the first and second images and the camera positions of the first and second cameras.

  In one embodiment of model reconstruction, a correspondence between image features of at least one portion of the vehicle in the first and second images is determined. The triangulation method can then be used to determine a model from image feature correspondence as well as the camera position of the first camera and the second camera. See, for example, reference [9].

  The model of the object at least describes depth information of at least a portion of the object, as described above. The model may further include, but is not limited to, one of the following attributes: shape, symmetry, planarity, geometric size, color, texture and density.

  The physical geometry of the at least one portion of the vehicle captured in the first and second images can be known, for example, restored as taught in reference [4] It will be possible to determine the exact scale factor for the model. The reconstructed model can be represented as 3D vertices, polygonal faces and / or edges spanned by these vertices. The sides and faces of the model may also be represented as splines or NURBS surfaces.

  The restored model may be determined in the vehicle coordinate system. The reconstructed model may also be determined in any coordinate system, such as its own coordinate system. In this case, the restored model can be associated with the vehicle coordinate system based on the position of at least one portion of the vehicle within the vehicle coordinate system.

  According to one embodiment, a third camera of the mobile device may capture a third image. The third image comprises a portion of the vehicle that is represented by at least a portion of the reconstructed model. The portion of the vehicle need not include at least one portion of the vehicle captured by the first camera, or only a portion thereof. In particular, the portion of the vehicle captured in the third image need not include the display of the vehicle or any visual content displayed on the display.

  The camera position of the third camera relative to the model while the third image is captured may be determined according to at least a portion of the reconstructed model and at least a portion of the third image. For example, the SLAM method (see reference [4]) could be used to determine the camera position of the third camera.

  In one example, the position of the third camera in the vehicle coordinate system while the third camera captures the third image is also used to determine the position associated with the mobile device in the vehicle coordinate system Good. The position associated with the mobile device may be the same as the position of the third camera in the vehicle coordinate system. The position associated with the mobile device in the vehicle coordinate system may have a spatial displacement from the position of the third camera in the vehicle coordinate system.

  The camera position of the third camera in the vehicle coordinate system can be determined from the camera position of the third camera relative to the model and the spatial relationship between the model and the vehicle coordinate system.

  Furthermore, the model tracks the camera (s) of the mobile device in the vehicle coordinate system and / or extends the reconstructed model by adding other reconstructed parts of the vehicle. It can further be used to initialize vision-based Simultaneous Localization and Mapping (SLAM) methods, such as those described in references [4, 5, 6].

  For example, a fourth image of at least one second portion of the vehicle is captured by a fourth camera of the mobile device. The fourth image may further capture a portion of the vehicle represented by at least a portion of the reconstructed model. Thereafter, the reconstructed model could be extended to have the first and second parts of the vehicle based on the SLAM method. This calculates the camera pose of the fourth camera relative to the model while capturing the fourth image based on matching the image positions of the vehicle portion and at least a portion of the reconstructed model of the portion of the vehicle You may need to

  The position associated with the mobile device can be determined according to the camera position of the mobile device's camera in the vehicle coordinate system. The camera position could be estimated according to at least a portion of the reconstructed model and at least one image captured by the camera based on the SLAM method. To this end, the at least one image captures a portion of the vehicle represented by at least a portion of the reconstructed model. The position associated with the mobile device may be the same as the position of the camera while the camera captures at least one image.

  The determined position or determined motion associated with the mobile device within the vehicle coordinate system may be provided to the vehicle to control at least one function of the vehicle according to the position or motion. At least one function may be associated with one or more subsystems of the vehicle. The vehicle subsystem could be, but is not limited to, an acoustic system, an ambient light system, and an air conditioning system.

  At least one function could control or configure one or more subsystems of the vehicle according to the determined position associated with the mobile device within the vehicle coordinate system. For example, the position could specify the central location of the space area that the ambient light system should illuminate, or the space area where the air conditioning system should monitor temperature. The listening area may also be designated by position so that the sound system will be controlled or adjusted to direct it towards the designated listening area.

  The mobile device may communicate with the vehicle directly or via a computer network. The mobile device may communicate indirectly with the vehicle via other devices, for example via a web services computer.

  A computer network may be a telecommunications network that connects processing devices (eg, computers) to enable communication and data exchange between systems, software applications, and users. The processing devices may be connected via a cable or wirelessly, or both via a cable and wirelessly. For example, the computer network could be the Internet, an intranet, a local area network, or a wide area network.

  In one embodiment, the mobile device may comprise one camera. The first camera, the second camera, the third camera, and the fourth camera are the same camera.

  In another embodiment, the mobile device may comprise at least two cameras. At least two of the first camera, the second camera, the third camera, and the fourth camera are different cameras.

  In another embodiment, at least two of the first camera, the second camera, the third camera, and the fourth camera are the same camera.

  Camera specific parameters for the first camera, the second camera, the third camera, and / or the fourth camera may be provided or calibrated based on a camera calibration method. The camera specific parameters may be used to determine camera position or camera motion based on the camera image.

  In one embodiment, a system according to the invention could be electronically coupled to or part of a mobile device. It could be separate from the vehicle. The system may communicate with the vehicle via a cable, wirelessly or via a computer network. The system could determine the position or movement associated with the mobile device within the vehicle coordinate system, and this position or movement could be transmitted to the vehicle to control at least one function.

  In a further embodiment, the system will be able to receive vehicle information from the vehicle. Vehicle information may include a vehicle coordinate system, the visual appearance of the displayed visual content, the resolution of the display used for display, the position of the display (ie at least one portion of the vehicle) within the vehicle coordinate system, the display It may include at least one of the physical geometry of the device (i.e., at least one portion of the vehicle), or a combination thereof.

  In another embodiment, the system could be electronically coupled to or part of the vehicle and be separate from the mobile device. The system may communicate with the mobile device via a cable, wirelessly or via a computer network. The mobile device will be able to transmit camera images to the system. The mobile device may further need to transmit camera specific parameters (e.g. focal length and principal point) to the system. The camera specific parameters may be used to determine the camera position based on the camera image.

  In another embodiment, the system could be separate from the mobile device and separate from the vehicle. The system may communicate with the mobile device and / or the vehicle via a cable, wirelessly or via a computer network. The system will be able to receive vehicle information from the vehicle and receive camera images and camera specific parameters from the mobile device. The system then determines the position or movement associated with the mobile device within the vehicle coordinate system, and controls the at least one function of the position or movement of the vehicle or of the vehicle associated with the at least one function. It could be sent to more than one subsystem.

  According to another aspect, the invention also relates to a computer program product comprising a software code section adapted to perform the method according to the invention. In particular, the computer program product is embodied on a computer readable medium and is non-transitory. The software code section may be loaded onto one or more memories of the processing device as described herein.

  According to one embodiment, the method may be implemented as an application executing on one or more processing devices of a mobile device, such as a mobile phone, in direct or indirect communication with a vehicle.

  According to another embodiment, the method may be implemented as an application executing on one or more processing devices of the vehicle and communicating directly or indirectly with the mobile device and / or the camera.

  According to another embodiment, the method runs on one or more processing devices of a computer, such as a mobile computer or a personal computer, and communicates directly or indirectly with mobile devices and / or cameras and vehicles. May be implemented as an application.

  Aspects and embodiments of the invention will now be described with reference to the drawings.

3 shows a flowchart of an exemplary embodiment of the present invention. Fig. 6 shows a flow chart of another exemplary embodiment of the present invention. 3 illustrates an exemplary scene according to an embodiment of the present invention.

  In the following, aspects and embodiments of the present invention will be described with reference to the exemplary embodiments as shown in FIGS. Although the following embodiments are described in connection with the use of a mobile phone, the invention applies in principle with any mobile device that is associated through a specific spatial relationship with the camera for capturing the image. Can.

  FIG. 3 illustrates an exemplary scene of a vehicle environment, wherein a portion of a vehicle 3001 includes a front control panel 3009 and a light source 3007. The display device includes a screen 3004 mounted on the front control panel 3009. Possible visual content displayed on the screen 3004 is a 2D map 3005, but may likewise be any other visual content. A vehicle coordinate system 3006 is associated with the vehicle 3001.

  In the exemplary scene shown in FIG. 3, the mobile device as used herein is a mobile phone 3002 comprising a camera 3003. Camera specific parameters of the camera 3003 are provided to the mobile device.

  In a possible use scenario, vehicle 3001 wirelessly transmits vehicle information to mobile phone 3002 via a computer server. Vehicle information includes at least a portion of the displayed 2D map 3005 image information (ie, the visual appearance), the resolution of the screen 3004, the position of the screen 3004 within the vehicle coordinate system 3006, and the shape and size of the screen 3004.

  In another embodiment, the screen 3004 may show the pictorial outline of the mobile phone 3002. The pictorial outline has a known position in the display coordinate system of the screen 3004 and therefore could also have a known position in the vehicle coordinate system 3006. If the camera 3003 has a known position relative to the mobile phone 3002, then when the mobile phone 3002 is placed on the position indicated by the pictorial outline on the screen 3004, the camera 3003 is in the vehicle coordinate system 3006. It would be possible to have a known initial position. This initial position of the camera 3003 in the vehicle coordinate system 3006 may be used to track the camera 3003 and / or restore a portion of the vehicle 3001. The position of the camera 3003 may be determined relative to its own initial position. That is, for example, the motion of the camera 3003 is determined based on an inertial sensor attached to the camera 3003 or based on corresponding image features between two images captured by the camera 3003. Therefore, the camera 3003 and thus the mobile phone 3002 associated with the camera 3003 can be tracked in the vehicle coordinate system 3006.

  The mobile device, here a mobile phone 3002, comprises at least one or more processing units, such as one or more microprocessors and associated circuitry. Processing devices are commonly used in the art and they are not shown in the figure because they are internal to the mobile phone 3002. In FIG. 3, reference numeral 3011 designates an internal processing unit. Among the tasks commonly used and applied in the art, in the context of the present invention, the processing unit 3011 communicates with the camera 3003 and the steps as described with reference to FIGS. 1 and 2 etc. Are configured to perform the tasks and steps as described herein in relation to the present invention. In this example, the mobile phone 3002 may be held by the user. The camera specific parameters of the camera 3003 may be provided to the processing unit 3011 or determined by a camera calibration procedure.

  Additionally, the vehicle 3001 may also include at least one or more processing units designated by reference numeral 3021 in FIG. 3, such as one or more microprocessors and associated circuitry generally used in the art. Good. Among the tasks commonly used and applied in the art, in the context of the present invention, the processing unit 3021 is configured to display visual content 3005 on a screen 3004. The processing unit 3021 is further applicable to perform tasks and steps as described herein in the context of the present invention, such as the steps as described with reference to FIGS. 1 and 2. .

  According to one embodiment, receiving image information associated with an image captured by a camera (such as camera 3003) directly from the camera or from another processing device, according to the received image information, within the vehicle coordinate system Steps according to the invention as described herein, such as determining the position associated with the mobile device and providing position information indicating the position associated with the mobile device in the vehicle coordinate system The processing units 3011 and / or 3021 may be suitable and configured for each, or any combination with each other, to perform any of the above. For example, such position information may be sent directly to the vehicle control equipment via direct communication or indirectly via another processing device (such as a server computer) to control at least one function of the vehicle according to the position. It may be provided.

  These tasks and steps are also not included in either the mobile phone 3002 or in the vehicle 3001, but in other devices such as the server computer 4000 that wirelessly communicates with the mobile phone 3002 and the vehicle 3001, for example, through a computer network. May be performed by one or more other processing devices, such as processing device 4001. Furthermore, all or part of the tasks and steps according to the invention as described herein may be shared or distributed among the processing units 3011, 3021 and 4001. .

  The camera 3003 and the housing of the mobile phone 3002 have fixed positions with respect to each other. The spatial relationship between the camera 3003 and any portion of the mobile phone 3002 may be provided by the manufacturer of the mobile phone 3002 or may be calibrated as commonly known in the art.

  Turning now to the process as described with reference to FIG. 1, in an exemplary embodiment of the invention as shown in FIG. 1, in step 1001, a vehicle provided with a vehicle coordinate system is provided. A mounted display displays a diagram, such as visual content in the form of a 2D map 3005 as shown in FIG. 3, on its screen. In step 1002, the vehicle transmits vehicle information from the vehicle to a mobile phone equipped with a camera via the Internet. At step 1003, the camera of the mobile phone captures an image of the figure displayed on the display device. In the exemplary scene shown in FIG. 3, the camera 3003 captures an image of the displayed 2D map 3005. The image may include only a portion of the displayed 2D map 3005.

  Step 1004 determines the camera position in the vehicle coordinate system while the camera captures an image, in this embodiment on the mobile phone. In the exemplary scene shown in FIG. 3, the camera position of the camera 3003 in the vehicle coordinate system 3006 can be determined while the camera 3003 captures an image. This is performed in accordance with the received image information associated with the captured image on the processing unit 3011 of the mobile phone 3002. The determination of the camera position could be based on matching the image captured by the camera 3003 and the image information of the 2D map 3005 received at the mobile phone 3002. From the determined camera position, the position associated with the mobile device 3002 in the vehicle coordinate system 3006 may be determined which may be used as an input to control at least one function of the vehicle. The known spatial relationship between the camera 3003 and any part of the mobile device 3002 may be used for this determination.

  Step 1005 transmits the determined camera position from the mobile phone to the vehicle. In the exemplary scene shown in FIG. 3, the camera position of the camera 3003 in the vehicle coordinate system 3006 is provided from the mobile phone 3002 to the vehicle 3001.

  Next, at step 1006, the vehicle may control at least one function of the vehicle in accordance with the received camera position. In the exemplary situation shown in FIG. 3, the processing unit 3021 of the vehicle 3001 may control the illumination direction of the light source 3007. The light source 3007 could be directed to illuminate an area 3008 along the direction defined by the camera position of the camera 3003.

  FIG. 2 shows a flow chart of another exemplary embodiment. Step 2001 displays the figure on a display mounted on a vehicle provided with a vehicle coordinate system. At step 2002, the vehicle sends vehicle information to a mobile phone equipped with a camera. At step 2003, the camera of the mobile phone captures a first image of the displayed figure. Step 2004 determines the camera position P1 in the vehicle coordinate system while the camera captures the first image according to the first image and the displayed figure on the mobile phone. In step 2005, the camera, ie the mobile phone, is moved to another position different from the position P1. At step 2006, the camera of the mobile phone captures a second image of the displayed figure. Step 2007 determines the camera position P2 in the vehicle coordinate system while the camera captures the second image, according to the second image and the displayed image information on the mobile phone. Step 2008 reconstructs a model of at least a portion of the vehicle on the mobile phone based on the image information of the first and second images and the camera positions P1 and P2. This could be achieved, for example, using a triangulation method, as disclosed in reference [9].

  At step 2009, the camera is moved. At step 2010, the camera captures a third image. Step 2011 determines the camera position P3 in the vehicle coordinate system while the camera captures the third image according to the model on the mobile phone. This could be implemented based on the SLAM method. See, for example, reference [4]. In step 2012, it is determined whether the position P3 is a desired position. The desired position indicates that the user desires to set or control the features of the vehicle according to the desired position. For example, the user may read a book. The desired position may indicate the central position of the area around the book to be illuminated by the vehicle's ambient light system. If it is not the desired position, then proceed to step 2008 and extend the model according to the third image, for example using the SLAM method (see, for example, reference [4]). On the other hand, if it is the desired position, then the position P3 is transmitted from the mobile phone to the vehicle (step 2013). Next, the vehicle may control at least one function of the vehicle according to the received position P3 (step 2014).

  The steps and system components as described above with regard to the first aspect of the method and system for providing position information for controlling at least one function of a vehicle are captured by at least one camera of a mobile device At least one of the vehicles using the same or similar components by having the modification of providing image information associated with the at least two images and the following steps taking into account this image information as described above It can be applied analogously with regard to the second aspect of the method and system for providing exercise information for controlling a function. Furthermore, components as described above, such as mobile devices, cameras, and / or processing devices may also provide the method of the second aspect for providing motion information to control at least one function of the vehicle. It may be configured to run.

  Throughout the specification, image information associated with an image is provided or received. This may include providing or receiving any processed or unprocessed information (version) of the image, portions of the image and / or features of the image, which allow position or pose estimation. Are known. The present invention does not require any provision or reception of raw image data. As such, processing may include compression (eg, JPEG, PNG, ZIP), encryption (eg, RSA encryption, Schnorr signature, El-Gamal encryption, PGP), conversion to another color space or grayscale, image Including cropping or scaling, or conversion to sparse representation based on feature descriptors, extraction, and any combination thereof. All these image processing methods can optionally be carried out and are covered by the term image information of the image or image information associated with the image.

Reference 1. U.S. Patent No. 8373763 (B1) 2. U.S. Patent Application Publication 2012/0287287 (A). International Publication No. 2011/075226 (A1) Issue 4. Davison, Andrew J. , Et al. "MonoSLAM: Real-time single camera SLAM." Pattern Analysis and Machine Intelligence, IEEE Transactions on 29.6 (2007): 1052-1067.
5. Strasdat, Hauke, J. et al. M. M. Montiel, and Andrew J. Davison. "Proceedings of Robotics: Science and Systems (RSS). Vol. 2. No. 3. 2010.
6. Lemaire, Thomas, et al. "Vision-based slam: Stereo and monocular approaches." International Journal of Computer Vision 74.3 (2007): 343-364.
7. U.S. Patent No. 7,660,437 (B1) 8. U.S. Patent No. 7,415,126 (B1) 9. Hartley, Richard, and Andrew Zisserman. Multiple view geometry in computer vision. Vol. 2. Cambridge, 2000.

Claims (25)

A method for providing position information for controlling at least one function of a vehicle (3001) provided with a vehicle coordinate system (3006), said method comprising
Receiving image information associated with at least one image of at least one portion of the vehicle (3004) captured by a camera (3003) of the mobile device (3002), wherein the mobile device is separate from the vehicle What you are doing,
Determining a position associated with the mobile device (3002) in the vehicle coordinate system (3006) according to the image information associated with the at least one image;
Providing position information indicating the position associated with the mobile device (3002) in the vehicle coordinate system (3006) to control at least one function of the vehicle according to the position;
Method including.   The method of claim 1, further comprising providing or receiving a position of the at least one portion of the vehicle (3004) within the vehicle coordinate system (3006).   The method according to claim 2, wherein the at least one portion of the vehicle comprises visual content (3005) displayed on a display (3004) mounted on the vehicle (3001). Determining the position associated with the mobile device (3002) within the vehicle coordinate system (3006),
Determining the camera position of the camera (3003) relative to the at least one portion of the vehicle (3004) in capturing the at least one image according to the image information associated with the at least one image;
Determine the position associated with the mobile device (3002) in the vehicle coordinate system (3006) according to the camera position in the vehicle coordinate system (3006) and the position of the at least one portion of the vehicle (3004) And
The method according to claim 2 or 3, comprising Determining the position associated with the mobile device (3002) within the vehicle coordinate system (3006) wherein the at least one image is a first image and the camera (3003) is a first camera But,
Receiving image information associated with a second image of the at least one portion of the vehicle (3004) captured by a second camera (3003) of the mobile device (3002);
Restoring a model of the at least one portion of the vehicle (3004) according to the image information associated with the first image and the second image;
The method according to claim 2 or 3, comprising Receiving image information associated with a third image captured by a third camera of the mobile device (3002), wherein the third image is represented by at least a portion of the reconstructed model Capturing at least a portion of the vehicle.
Determining a camera position of the third camera relative to the reconstructed model in capturing the third image according to the at least a portion of the reconstructed model and image information associated with the third image When,
Determining the position associated with the mobile device (3002) in the vehicle coordinate system (3006) according to the camera position of the third camera;
The method of claim 5, further comprising Determining the camera position of the first camera (3003) relative to the at least one portion of the vehicle (3004) when capturing the first image according to at least a portion of the image information associated with the first image And
Determining the camera position of the second camera relative to the at least one portion of the vehicle (3004) in capturing the second image according to at least a portion of the image information associated with the second image; ,
The method according to claim 5 or 6, further comprising The at least one portion of the vehicle (3004) is a first portion of the vehicle;
Receiving image information associated with a fourth image captured by a fourth camera of the mobile device (3002), the fourth image being at least a portion of the first portion of the vehicle And capturing at least a second portion of the vehicle.
Expanding the restored model to include the first portion of the vehicle and the second portion of the vehicle;
The method according to any one of claims 5 to 7, further comprising   The method according to any one of claims 5 to 8, wherein the reconstructed model at least describes depth information of at least a portion of the vehicle.   10. The device according to any of claims 5 to 9, wherein at least two of the first camera, the second camera, the third camera, and the fourth camera are the same camera (3003). the method of.   The method according to any one of claims 5 to 9, wherein at least two of the first camera, the second camera, the third camera and the fourth camera are different cameras.   The system according to claim 1, wherein the at least one function is associated with at least one of an acoustic system of the vehicle, an ambient light system of the vehicle, and an air conditioning system of the vehicle and controlled according to the position. The method according to any one of 11. A method for providing motion information for controlling at least one function of a vehicle (3001) provided with a vehicle coordinate system (3006), said method comprising
Receiving image information associated with at least two images captured by at least one camera (3003) of the mobile device (3002), the mobile device being separate from the vehicle, the at least two Each of the images capturing at least one portion of the vehicle;
Determining movement associated with the mobile device (3002) relative to the vehicle coordinate system (3006) according to the image information associated with the at least two images;
Providing exercise information indicative of the movement associated with the mobile device relative to the vehicle coordinate system (3006) to control at least one function of the vehicle according to the movement;
Method including.   The method according to claim 13, wherein the movement associated with the mobile device (3002) relative to the vehicle coordinate system (3006) is determined according to at least two positions associated with the mobile device within the vehicle coordinate system.   The movement associated with the mobile device (3002) relative to the vehicle coordinate system (3006) is determined according to at least two camera positions of the at least one camera (3003) in capturing the at least two images. A method according to claim 13 or 14.   The movement associated with the mobile device (3002) relative to the vehicle coordinate system (3006) is according to the image position of the at least one portion of the vehicle (3004) captured inside the each of the at least two images. The method according to any one of claims 13 to 15, which is determined.   Among the at least one function controlled based on the determined movement associated with the mobile device, in particular an acoustic system, a lighting system and an air conditioning of the vehicle controlled according to the movement of the mobile device The method according to any one of claims 13 to 16, wherein the function is associated with at least one of A system for providing position information for controlling at least one function of a vehicle (3001) provided with a vehicle coordinate system (3006), the system comprising:
A first processing unit (3011) configured to receive image information associated with at least one image of at least one portion of the vehicle (3004) captured by a camera (3003) of the mobile device (3002). 3021, 4001), the first processing unit (3011, 3021, 4001), wherein the mobile device is separate from the vehicle;
The first processing unit (3011, 3021) configured to determine a position associated with the mobile device (3002) in the vehicle coordinate system (3006) according to the image information associated with the at least one image. 4001),
Configured to provide position information indicating the position associated with the mobile device (3002) in the vehicle coordinate system (3006) to control at least one function of the vehicle (3001) according to the position A second processing unit (3011, 3021, 4001)
System with A system for providing motion information for controlling at least one function of a vehicle (3001) provided with a vehicle coordinate system (3006), the system comprising:
A first processing unit (3011, 3021, 4001) configured to receive image information associated with at least two images captured by at least one camera (3003) of the mobile device (3002), A first processing unit (3011, 3021, 4001), wherein the mobile device is separate from the vehicle, each of the at least two images capturing at least one portion of the vehicle;
The first processing unit (3011, 3021, 4001) configured to determine movement associated with the mobile device (3002) relative to the vehicle coordinate system (3006) according to the image information associated with the at least two images. )When,
A second processing unit configured to provide movement information indicative of the movement associated with the mobile device relative to the vehicle coordinate system (3006) to control at least one function of the vehicle according to the movement ( 3011, 3021, 4001),
System with   The system according to claim 18 or 19, wherein the first processing unit and the second processing unit are the same processing unit (3011, 3021, 4001).   21. A system according to any one of claims 18 to 20, wherein at least one of the first processing unit and the second processing unit (3011) is comprised in the mobile device (3002).   22. A system according to any one of claims 18 to 21, wherein at least one of the first processing unit and the second processing unit (3021) is comprised in the vehicle (3001).   The computer device (4000) in communication with the mobile device (3002) and the vehicle (3001) at least one of the first processing device and the second processing device (4001) 22. A system according to any one of 22.   The system according to any of claims 18 to 23, wherein the mobile device (3002) is a handheld device, in particular a mobile phone, a tablet computer or a mobile computer.   A computer program product comprising a software code section adapted to perform the method according to any one of claims 1 to 17 when loaded into the internal memory of a processing unit.

Images