EP2652680A1

EP2652680A1 - System and method for assisting the performance of a maintenance or operating process

Info

Publication number: EP2652680A1
Application number: EP10796006.4A
Authority: EP
Inventors: Sascha Skorupa; Martin Schleyer
Original assignee: Incca GmbH
Current assignee: Incca GmbH
Priority date: 2010-12-13
Filing date: 2010-12-13
Publication date: 2013-10-23
Also published as: WO2012079617A1

Abstract

The invention relates to a system for assisting the performance of a maintenance and/or operating process on a target device (30), comprising : an input unit (110) for selecting a maintenance and/or operating process and/or a component (32) of the target device (30) from a list of maintenance and/or operating processes and/or components (32) stored in a database (13, 130); a readout unit (130) for reading out contour data (Kon1) of the selected component (32) or a component (32) associated with the maintenance and/or operating process; a camera (105) for recording a plurality of images (Img1); an analyser (140) for analysing the images (Img1) and for identifying an image of the component (32) with reference to the contour data (Kon1) in the images (Img1); a labelling unit (150) for labelling the identified image of the component (32); a display device (200) for displaying at least one of the recorded images in conjunction with the labelling by the labelling unit (150). An essential idea of the invention may be seen in the fact that for the user of the system it is considerably easier to locate the selected component.

Description

MEISSNER BOLTE M / INX-015-PC

DEVICE AND METHOD FOR SUPPORTING THE PERFORMANCE OF A MAINTENANCE AND / OR OPERATION PROCESS

description

The invention relates to a device for supporting the performance of a maintenance and / or operating procedure on a target device, a method for

Performing a maintenance and / or operation on a target device and a computer-readable medium with program code for execution on a computing unit, which causes the arithmetic unit to perform said method.

There are known mobile phones with particularly powerful processors. Such mobile phones are referred to as smart phones, which significantly expand the functionality of a mobile phone. Advanced smartphones can over

additional programs, eg. B. applications, are equipped by the user individually with new features. Usually, a smartphone has

at least one input device, eg. As a keyboard or a touch screen, and at least one output device, for. As an LCD screen. Other facilities such as a speaker, a GPS receiver, etc. may be provided.

It is also known to provide such smartphones with applications that make it easier for the user to access specific addresses or locations or paths

find. The smartphones then take over the function of a

Navigation system, wherein a current position is detected, for example, using a GPS receiver and the route guidance by means of an output device, such as the display device or a speaker is made.

A use of such smartphones to support the implementation of a maintenance and / or operation is not known.

Based on this prior art, it is an object of the present invention to provide a device for supporting the performance of a maintenance and / or operation on a target device. Furthermore, a method for MEISSNER BOLTE M / INX-015-PC

2

Perform a maintenance and / or operation on a target device using a smartphone.

This object is achieved by a device according to the present claim 1, a method according to the present claim 9 and a computer-readable medium according to the claim 16.

In particular, the object is achieved by a device, in particular a smartphone, which has an input device for selecting a maintenance and / or

Operation and / or a component of the target device from a list of stored in a database maintenance and / or operating procedures and / or components:

- A read-out device for reading contour data of the selected

Component or one of the maintenance and / or operation

associated ingredient;

a camera for capturing a plurality of images;

an analyzing means for analyzing the images and identifying an image of the constituent from the contour data in the images;

- a marking device to identify the identified

Picture of the component;

- A display device for displaying at least one of the captured images in conjunction with the identification of the marking device.

The mentioned target devices may preferably be a motor or an engine compartment. On the other hand, it is also conceivable to understand by this a coffee machine or any other technical or non-technical device. The device according to the invention makes it possible to substantially facilitate maintenance and / or operating processes on these target devices. In this case, the user is in particular enabled to find essential components for the maintenance and / or operating processes which are integrally connected to or belong to the target device. MEISSNER BOLTE M / INX-015-PC

3

According to the invention, an input device, for example a touchscreen, a keyboard or any other operating element, is used to control a

certain maintenance and / or operation and / or a part of the

Select target device. As far as one decides for the selection of a maintenance and / or operation, these are usually assigned components that

must be found. The readout device determines stored

Contour data of the selected or assigned component. Thereafter, it is possible for the user to capture images preferably from the target device with the camera, which then in turn can be evaluated by the analysis device. The analysis device makes it possible to identify images of the component in the images on the basis of the contour data. The marking device identifies the identified picture or partial picture of the component. The display device then displays at least one of the captured images in which the image of the

Ingredient is marked so that the user knows which part it is.

For example, the user can select the maintenance procedure "oil change" by means of the input device, wherein the device according to the invention the

For example, user facilitates locating the closure part of the oil filler neck on the engine. Although in the above example the input device receives an input from the user, it may be configured to receive any data from other devices or programs. This data can for example specify a maintenance and / or operating procedure or a specific component.

In one embodiment, the device comprises a

Position determining device for detecting the orientation of the camera with respect to the target device and optionally the position of the camera with respect to the target device and a normalization device for imaging the images on a, preferably predetermined by the contour data, normalization plane. Since it is at the

Target devices and the components are mostly three-dimensional objects, but conventional cameras can only make two-dimensional recordings, it is extremely problematic to quickly identify pictures of the sought ingredient - preferably in real time - within the captured images. The device can therefore have a position-determining device which determines an orientation of the camera and optionally a position thereof relative to the target device. The exact or approximate actual orientation and / or actual positioning MEISSNER BOLTE M / INX-015-PC

4 helps to synchronize the existing contour data with the acquired image data more quickly. Preferably, the captured image data is normalized to lie in a predetermined normalization plane. The recorded images can thus be transformed in such a way that their representation corresponds to a specific orientation or positioning of the camera. Preferably, this orientation is predetermined on the basis of a normalization plane, for example in conjunction with the contour data. In this respect, it can be assumed that the

existing contour data were essentially captured from a similar recording direction, as the images shown. This allows faster finding of possible matches.

The navigation device can be designed to set a desired position and / or a desired orientation of the camera (for better detection of the target device,

in particular of the selected ingredient) and / or pretending. The navigation device can thus help to align the device, in particular the smartphone, in such a way that the camera can capture images containing the component to be found. Alternatively, it is also possible that the

Navigation device automatically aligns the camera accordingly.

The apparatus may comprise a storage means for storing the database containing a plurality of components of target devices and associated ones

Contour data includes. It is conceivable the necessary data from external

To query and receive data sources. Preferably, the device has a local memory containing necessary data. This data may include the contour data, the normalization level, and / or the target device. It is also conceivable to organize the data hierarchically depending on a selected target device.

The device may include a communication device for polling

Information, in particular additional information and / or contour data, include from a server. By the communication device, for example, the above contour data, relationships between maintenance or

Operations and components and / or lists of target devices are queried. On the other hand, it is also possible to retrieve additional information from an external data source to display it to the user in an appropriate manner.

For example, this additional information may be information that relates to the MEISSNER BOLTE M / INX-015-PC

Refer to 5 selected maintenance and / or operation. Preferably, the additional information indicates how the selected component is to be handled.

The already mentioned position determination device can be used to detect the

Orientation of the camera with respect to the target device and / or the position of the camera with respect to the target device to be implemented to fulfill this task for the target device or a part of the target device specified color pattern and / or a color distribution and then read by comparing the

Color distribution pattern with the captured images the desired data are determined. So it is not absolutely necessary that sensors, for example

Tilt sensors or a GPS receiver are provided to detect the actual alignment data and actual position data. The actual alignment data and actual position data of the camera can also be determined by means of image processing steps.

The position determination device can be designed to read out contour data of a multiplicity of different components assigned to the target device and to generate actual alignment data and actual position data of the camera on the basis of an alignment of the contour data with the captured images.

The device may comprise an additional information device, which

is configured to display additional information on the display device,

in particular if the mapping of the constituent has been identified by the analyzer. It is thus possible to provide additional information that supplement, for example, the maintenance and / or operation. Specifically, it can be specified how to deal with a particular ingredient that has been detected.

Furthermore, as already described, the object is achieved by a method for carrying out a maintenance and / or operating procedure on a target device using a smartphone, the method comprising the following steps:

- setting the target device;

- Selection of a maintenance and / or operating procedure and / or a

Component of the target device from a list in a database MEISSNER BOLTE M / INX-015-PC

6 stored maintenance and / or operations and / or components;

Detecting a plurality of images by means of at least one camera of the

smartphones;

- Reading out contour data of the selected component or a component

Maintenance and / or operation associated with component;

- Analyzing the images by the smartphone to identify a

Image of the component based on the contour data;

Displaying at least one image of the captured plurality of images on a display device of the smartphone, wherein an image of the component is identified when the image of the component is determined on the basis of the

Contour data has been identified.

The process results in similar advantages as they have already been described in connection with the device.

Further advantageous embodiments will become apparent from the dependent claims.

The invention will be described with reference to several embodiments, which are explained in more detail by means of illustrations. Here is a front view of a smartphone with display and

Input means;

FIG. 2 is a rear view of the smartphone of FIG. 1; FIG.

3 is a schematic representation of a read-out device,

Analyzing device, marking device,

Position determining device, normalizing device,

Navigation device, communication device;

4 shows an exemplary image representation of the display device

Fig. 1 is displayed; MEISSNER BOLTE M / INX-015-PC

7

Fig. 5 is a schematic illustration of image processing steps performed by the smartphone of Fig. 1; and

6 shows a server for communication with the smartphone from FIG. 1.

In the following description, the same reference numerals are used for the same and like parts.

1 shows a smartphone 100 according to the invention in a front view. The

Smartphone includes an antenna 191, a display device 200 and a

Input device 110, which has a plurality of keys 111, 11, by means of which a user can make inputs. The display device 200 serves to output information. Furthermore, the smartphone 100 has a processor 101 for processing programs and a memory device 120. The

Memory device 120 may be used to program code from

save programs to be executed and / or necessary data,

For example, configuration data to secure permanently. In the described

In the embodiment, the storage device 120 includes data for one

Device database 123.

Fig. 2 shows a rear view of the smartphone 100 of Fig. 1. The back of the smartphone 100 has a camera 105 for capturing image data Img l. The

Smartphone 100 of FIG. 2 is oriented such that a schematically illustrated motor 30 can be detected as a target device by means of the camera 105. The engine 30 has an oil filler neck 31 and a closure part 33.

In an exemplary embodiment, an application that prompts the user, such as one, runs on the described smartphone 100

Maintenance process for his car. The user may, among other things, select the "refill oil" operation, and then the smartphone 100 will search the appropriate database maintenance process 123 and determine that oil refilling is initially required to unlock the closure portion 32 of the oil filler neck 31 on the motor 30 Thus, the application prompts the user to open the hood and point his smartphone 100 at the engine compartment, then the application starts to capture image data Img 1 from the engine compartment, possibly including images of the engine 30 as shown in FIG. MEISSNER BOLTE M / INX-015-PC

8 record. The recorded image data Img l are displayed directly on the display device 200.

4 shows by way of example a representation of the image data Img l on the

Display device 200. According to the invention, the current application recognizes whether or not the image data Imgl includes an image of the shutter member 32, and then designates this image. In the embodiment shown in Fig. 4, the figure is characterized in that arrows 201, 20 point to the sought component, namely the closure member 32. In another embodiment, the contours of the sought component may be highlighted, for example, by a particular color. Also it would be conceivable, the sought after component

encircle.

After locating the sought-after component, the application preferably issues instructions that allow the user to perform the selected maintenance operation. For example, the user may be prompted to unscrew the indicated closure member 32, indicated by arrows 201, 20, and refill oil as needed.

To ensure the described procedure, the processor 101 of the smartphone 100 implements a plurality of devices, which are shown schematically in FIG. 3

are shown. This includes a read-out device 130 for reading out the contour data Koni from the device database 123, an analysis device 140 for analyzing the recorded image data Imgl, a marking device 150 for identifying the identified image of the component. Optionally, a normalization device 170 for normalizing the acquired image data Imgl, a navigation device 180 for finding a suitable orientation of the camera 105 and a communication device 190 may be provided.

By way of example, FIG. 5 illustrates a possible image processing process which is used to identify the sought-after component-in the exemplary embodiment

Closing part 32 - passes through the marking device 150. Thus, as already stated, a plurality of image data Img l from the camera 105

recorded, which are respectively supplied to the analysis device 140. Of

Further, on the basis of the user by means of the input device 110th

inputs selected a maintenance operation associated with one or more components. For at least one of these ingredients MEISSNER BOLTE M / INX-015-PC

For example, contour data Koni are read from the device database 123 via an identification number ID from the read-out device 130. This contour data Koni can be vector data that maps the sought-after component in three-dimensional space with sufficient accuracy. On the other hand, this can also be merely two-dimensional image data which shows the actually three-dimensional object, namely the closure part 32, from a given view in a (normalized) image. The contour data Koni are also processed by the analysis device 140 and compared with the image data Imgl. As soon as the

Analyzer 140 within the image data Imgl discovers an image corresponding to the contour data Koni of the searched constituent, corresponding information is output to the identifier 150, which then performs an identification (eg, the label shown in Fig. 4).

It should be clear to the expert working here that there are a variety of

Provides image processing capabilities to identify the image of the shutter portion 32 within the image data Img l. So it would be theoretically possible to create a three-dimensional model of the entire engine compartment, to extract suitable contours from the image data Imgl and to add them to the model in this way

Compare that it can be determined which part of the engine compartment is currently detected by the camera 105. On the other hand, a comparison can take place in the two-dimensional image space. In a preferred embodiment, the contour data Koni are present in two dimensions. The captured image data Img l are using the

Normalizing means 170 shown such that the application can assume that as far as the closure member 32 is detected by the camera 105, there is a sufficient similarity between the stored contour data Koni and the extracted contours.

Preferably, the normalization device 170 at least roughly knows the orientation and / or the position of the camera 105 with regard to the target device, for example the motor 30. The device database 123 can be used with respect to the

Contour data Koni specify a normalization level. The

Normalization device 170 then determines to what extent the actual orientation of camera 105 deviates from a desired orientation of camera 105 and performs suitable transformations so that image data Img 1 can be compared with contour data Koni. The actual orientation of the camera 105 may be adjusted by suitable sensors, e.g. As inclination sensors are detected. On the other hand, it is conceivable to make assumptions in this regard. For example, it is without an engine room MEISSNER BOLTE M / INX-015-PC

10 considerable effort only possible to detect the motor 30 from a particular direction. This information may be sufficient to store the image data Img l in

to normalize appropriately.

On the other hand, the recorded image data Imgl can be used to determine the actual position and / or the actual orientation of the camera 105 in an iterative process. Thus, with respect to the engine compartment certain color patterns

stored, which allow a rough determination of the position in a simple manner. So the usual components of an engine compartment are gray or black. Only a few areas differ significantly from these predominantly dark colors. For example, there may be a blue cap for the fountain solution and a bright car battery. The car battery may be located on one side of the engine compartment while the blue wiper cap is on the other side.

Once the camera 105 detects one or both of these components, it is relatively easy to calculate a position and / or orientation of the camera 105.

This position can then be used to process the image data Img l for finding the contour data Koni.

Furthermore, this information obtained with respect to the actual position and / or the actual orientation of the camera 105 can be used to indicate to the user instructions on how to align the camera in a suitable manner to the target device - in the embodiment, the motor 30 -. and / or the sought-after component - in the exemplary embodiment, the closure part 32 - to capture. For example, in the illustration of the display device 200 shown in FIG. 4, arrows may appear at the edges that instruct the user to move the smartphone 100 to the left, right, up, or down. For example, in the aforementioned example, the camera 105 may detect only the battery. The navigation device 180 knows that the closure part 32 is located to the right of the battery. In this respect, it can output corresponding instructions so that the user moves the smartphone 100 to the right until the image data Imgl contains an image of the closure part 32. On the other hand

corresponding instructions are output via a loudspeaker.

The professional working here knows a variety of ways, such as

Position determining means 160 based on acquired image data coarse actual positions and / or actual orientations of the camera 105 can detect. Furthermore, he knows how the navigation device 180 has to process this data to the MEISSNER BOLTE M / INX-015-PC

11

User target positions and / or target orientations of the camera 105 or the

Pretend smartphones 100.

In the embodiment described above, all data related to

Detection of the searched component and to select the maintenance and / or operation are necessary stored on the storage device 120.

Theoretically, however, it would also be possible to interrogate this data at least partially from a remote device, for example a server 10 (see Fig. 6).

This polling operation may be performed by the communication device 190, which establishes a communication link to the server 10. For this purpose, the server 10 has a transmitting / receiving unit. Furthermore, the server has a server database 13, which may have a similar structure to the one

Device database 123 of the smartphone 100. Herein, for example, a

Identification number ID, contour data Koni and additional information Data

get saved. The smartphone, in particular the communication device 190, can interrogate the necessary data interactively from the server 10 and make it easier to find the component with it.

In another embodiment, a portion of the data is local to the

Smartphone 100 saved. Further data, such as additional information Data are queried by the server 10. Preferably, these data can at least temporarily be temporarily stored on the smartphone 100, in particular on the memory device 120.

In the described embodiment, the keys 111, 11 are used to select a maintenance and / or operating procedure and / or a component. Theoretically, this selection could also be made using a touchscreen. Furthermore, this selection can be made by another application.

Likewise, the application described can be designed to find components of the target device independently of a maintenance and / or operating procedure. For example, the user can directly input that he is looking for the closure part 32 of the oil filler neck 31. An indication that he may be oil

does not have to be refilled.

In the presently described embodiment, the target device was a motor or engine compartment. It should be clear that the target device is any technical MEISSNER BOLTE M / INX-015-PC

12

Device, such as a table fan, a coffee maker, a bicycle, or a chair can be. Likewise, the most diverse constellations with respect to the sought ingredient are conceivable. This can be, for example, a

Switch, a screw, a connecting cable, etc. act.

It is conceivable to implement an additional information device which

Additional Information Data outputs as soon as the sought-after component is found. This additional information can be in any format, eg. As image data, text data, video data, sound data, present and output in a suitable manner.

LIST OF REFERENCES

10 servers

11 Transceiver unit

13 server database

30 engine

31 oil filler neck

32 closure part

100 smartphone

101 processor

105 camera

110 input device

111, 111 'button

120 storage device

123 device database

130 read-out device

140 analysis device

150 marking device

160 position determination device

170 normalizing device

180 navigation device

190 communication device

191 antenna

200 display device

201, 201 'arrow

ID identification number

Imgl image data MEISSNER BOLTE M / INX-015-

13

contour data

Further information

Claims

MEISSNER BOLTE M / INX-015-PC Device for supporting the performance of a maintenance and / or operating procedure, corresponding method and computer-readable medium with program code for executing a corresponding method

1. Device, in particular smartphone (100), in support of

Carrying out a maintenance and / or operating procedure on a target device (30), comprising:

- An input device (110) for selecting a maintenance and / or

Operating procedure and / or a component (32) of the target device (30) from a list of maintenance and / or operating processes and / or components (32) stored in a database (13, 130);

- a read-out device (130) for reading out contour data (Koni) of the selected component (32) or of a component (32) assigned to the maintenance and / or operating process;

a camera (105) for capturing a plurality of images (Imgl);

an analysis device (140) for analyzing the images (Imgl) and for identifying an image of the component (32) on the basis of the contour data (Koni) in the images (Imgl);

- a marking device (150) for identifying the identified image of the component (32);

- A display device (200) for displaying at least one of the captured images in conjunction with the identification of the marking device (150).

2. Apparatus according to claim 1,

marked by

a position determination device (160) for detecting the orientation of the camera (105) with respect to the target device (30) and optionally the position of the camera (105) with respect to the target device (30);

a normalization device (170) for imaging the images (Img l) on a, preferably by the contour data (Koni) predetermined,

Normalization level. MEISSNER BOLTE M / INX-015-PC

2

3. Device according to one of the preceding claims,

marked by

a navigation device (180) for receiving actual orientation data of the camera (105) with respect to the target device (30) and actual position data of the camera (105) with respect to the target device (30) and for outputting a desired position and / or a target Orientation of the camera (105) for detecting the target device (30), in particular the selected component (32).

4. Device according to one of the preceding claims,

marked by

a storage means (120) for storing the database (123) containing a plurality of components (32) of target devices (30) and associated ones

Contour data includes.

5. Device according to one of the preceding claims,

marked by

a communication device (190) for querying information, in particular additional information and / or contour data (Koni), from a server (10).

6. Device according to one of the preceding claims,

d a d u rc h e ke n ze i c h n et that

a position determining device (160) is designed to detect the orientation of the camera (105) with respect to the target device (30) and the position of the camera (105) with respect to the target device (30)

to read in a color distribution pattern specific to the target device (30) or a part of the target device (30), and

to produce actual alignment data and actual position data of the camera (105) on the basis of a comparison of the color distribution pattern with the captured images (Imgl).

7. Device according to one of the preceding claims,

d a d u rc h e ke n ze i c h n et that

a positioning device (160) for detecting the orientation of the camera (105) with respect to the target device (30) and the position of the camera (105) with respect to the target device (30) is adapted to contour data (Koni) a plurality of different, the Target device (30) associated MEISSNER BOLTE M / INX-015-PC

3

Components (32) read and based on a comparison of the contour data (Koni) with the captured images (Imgl) actual alignment data and actual position data of the camera (105) to produce.

8. Device according to one of the preceding claims,

marked by

an additional information device which is designed for this purpose

To display additional information on the display device (200),

in particular, if the mapping of the constituent has been identified by the analyzer (140).

9. A method for performing a maintenance and / or operation on a target device (30) using a smartphone (100),

in particular of a smartphone (100) according to one of claims 1 to 8, comprising the steps:

- setting the target device (30);

- Selection of a maintenance and / or operating procedure and / or a

Component (32) of the target device (30) from a list of maintenance and / or operator actions and / or components (32) stored in a database;

- Capturing a plurality of images (Imgl) by means of at least one camera (105) of the smartphone (100);

- reading contour data (Koni) of the selected component (32) or of a component (32) assigned to the maintenance and / or operating procedure;

- Analyzing the images (Img l) by the smartphone (100) for identifying an image of the component (32) based on the contour data (Koni);

Displaying at least one image of the acquired plurality of images (Imgl) on a display device (200) of the smartphone (100), wherein an image of the component (32) is identified if the image of the component (32) is based on the contour data (Koni) was identified.

10. The method according to claim 9,

marked by

- detecting an orientation and / or a position of the camera (105) with respect to the target device (30); MEISSNER BOLTE M / INX-015-PC

4

- Mapping the plurality of images (Imgl) on a, preferably predetermined by the contour data, normalization under

Considering the detected orientation and / or position of the camera (105) with respect to the target device (30).

11. The method according to claim 9 or 10,

marked by

- Detecting an actual orientation and / or an actual position of the camera (105) relative to the target device (30);

- issuing guidance instructions regarding a target position

and / or a target orientation of the camera (105) for detecting the target device (30), in particular of the selected component (32).

12. The method according to any one of claims 9 to 11,

marked by

querying information, in particular additional information and / or contour data (Koni), from a server (10) via a preferably wireless communication connection.

13. The method according to any one of claims 9 to 12,

d a d u rc h e ke n ze i c h n et that

detecting an actual orientation and / or an actual position of the camera (105) with respect to the target device (30) by reading in a target specific to the target device (30) or a part of the target device (30)

Color distribution pattern and on the basis of a balance of the color distribution pattern with the captured images (Imgl) takes place.

14. The method according to any one of claims 8 to 12,

d a d u rc h e ke n ze i c h n et that

a / the detection of an actual orientation and / or an actual position of the camera (105) with respect to the target device (30) based on a readout of contour data (Koni) a plurality of different, the target device (30) associated components (32) and Based on a matching of the contour data (Koni) with the captured images (Imgl). MEISSNER BOLTE M / INX-015-PC

5

15. Method / device according to one of the preceding claims,

d a d u rc h e ke n ze i c h n et that

specify the contour data (Koni) a 2-D contour.

16. Computer-readable medium with program code for execution on one

Arithmetic unit (101) which causes the arithmetic unit (101) to perform the method according to one of claims 9 to 13.