DE102022003877A1 - Mobile terminal for use in automated building locks, and a method for object-oriented use of a mobile terminal in automated building locks - Google Patents

Abstract

The invention relates to a device for diagnostics and / or commissioning and / or assembly of building locks, which are equipped with a drive device for opening and closing, the device from a mobile augmented reality (AR) - capable terminal consists, which in addition to a real image of the building closure can also display at least one virtual image or image components of the building closure with the drive device on a screen, for which purpose the AR terminal has a camera for actually recording an existing building closure in or on a building, as well as a storage means for storing documentation data different building lock versions, which are stored in different programs within the AR terminal and can be processed by a data processing unit with the processing of the real image components recorded by the camera, a control panel for communication with the drive device of the real building lock, and a data connection device between the drive device a wireless data transmission between the driving device of the real building shutter and the mobile AR terminal of the real building shutter and the AR terminal. The invention also includes a method for using such a device with an AR-capable end device.

Description

The invention relates to a device consisting of a mobile augmented reality (AR)-capable end device for use in object-related building locks for troubleshooting and/or maintenance and/or commissioning of building locks, which are equipped with a drive device for opening and closing, as well as a method for using such a terminal in building locks.

With the US 6,6225,299 B1 a tracking system for determining the relative position between a sensor and an object surface has become known. This system includes a sensor for detecting a reference pattern arranged on an object surface and a processor connected to at least one sensor. This may be referred to as an augmented reality system, which generally includes a pattern of fiducial marks placed on a surface, a computer having a processor and memory, a user interface for receiving input, and presenting augmented reality Outputs for a user and a tracker to capture the fiducials.

A method for operating an augmented reality system is the DE 10 2015 215 613 A1 again. To generate an augmented reality image, a real image of a real object is recorded using a real camera and an edge image of the real object is generated from the real image of the real object, with the position of a virtual image component being recorded using the real camera Image is determined by means of the edge image, and the virtual image component with at least part of the real image becomes an augmented reality image, with an initialization edge image of the real object depending on the pose of the real object from the perspective of the real camera being automatically generated beforehand becomes.

The processing of registration marks is not fully applicable to building locks. The object of the invention is therefore to provide a device and a method, which should simplify the commissioning and troubleshooting and maintenance of an automated building lock.

The object is achieved by the features of a device according to claim 1 and a method according to claim 12. The dependent claims following the respective main claims reflect a further embodiment of the idea according to the invention.

In the following description and the claims, reference is generally made to an automated building lock, with a building lock comprising an automated gate and an automated door, so to speak. The invention essentially consists of three components, namely a building closure, a drive device that interacts with the building closure, and a mobile augmented reality (AR)-capable end device.

The term augmented reality (AR) expands reality so that the actual building closure viewed is enriched with virtual content through stored information. This means that additional information or data in real time expands the context of the building closure in question. This combination is perceived by the user as useful information. The immersive tasks that are immediately available are perceived as very informative by the user of a device. Thus, it is possible to carry out building lock settings and building lock operations guided by AR. This can be done using an AR-enabled end device in the form of a smartphone, tablet or similar device that is equipped with a camera. The camera is always aimed directly at the relevant building closure while using the AR device. The building closure can be set and operated in a live image with the drive device on the AR-enabled end device, or building closure information such as status, errors, etc. of the building closure can be displayed and called up. In addition, guided maintenance processes can be carried out directly on site. The IDs of the people using the end device enable the work carried out and the logging to be traced.

This can be done using an AR-capable end device in the form of a smartphone, tablet or the like, for example, which is equipped with a camera. The camera is aimed directly and constantly at the relevant building closure while using the AR device. The building lock can be set and operated in a live image on the AR-capable end device, or information such as status, errors, etc. of the corresponding building lock and the drive unit can also be displayed and read from the building lock. In addition, guided maintenance processes can be carried out directly on site. The IDs of the people using the AR terminal enable the work carried out and the logging to be traced. That's what the drive mechanism is for connected to the AR-enabled device via a wireless data connection, for example Bluetooth, WLAN, Smarthome or equivalent. This enables the AR-enabled device to provide information about the drive device, such as software versions, and to issue commands via a program. Such commands can be: open, close, stop, change settings or the like.

The AR-enabled device establishes an optical connection with the building closure, with the camera integrated in the AR device capturing the door in its entirety, preferably from the inside, including the design and attachments, and displaying it as a live image on the screen. In order to be able to do this, the building closure must be permanently and uninterruptedly recognizable in its entirety, whereby the camera of the AR device must be constantly pointed at the building closure by the user.

At least one program for installation or maintenance and processing with information of the building closure is stored on the AR terminal. Such a program works, for example, via a control panel with simultaneous information output. The camera acts as an eye and must therefore be aimed at the relevant building closure at all times. An image recognition program can be used to identify the building closure with the drive device without any doubt. The imaging of the building closure will preferably be done from the inside, i.e. from the building side view, but this can also be done from the outside.

After the installation of the building lock, a connection between the end device and the drive device is carried out by a wireless data connection, for example by the Bluetooth connection or WLAN or a smart home environment. By pointing the AR terminal at the building closure, the AR terminal can also simultaneously receive information from the driving device via the building closure via the wireless data link from the driving device. For example, an ID number of the building lock or of the automatic drive used to operate the building lock can be used as a unique identification. A gradual commissioning of the building closure with the drive device is significantly simplified.

If the drive device itself is not able to set up such a wireless data connection to the AR terminal, the drive device can also be retrofitted with an additional device, for example, so that even with older building locking systems or drive devices, such a wireless connection can be established via the AR terminal. terminal can be executed. Corresponding software in the form of different programs can thus be installed in the microprocessor-controlled drive device or on the additional device that is retrofitted at a later point in time.

Different program executions are stored on the AR-enabled device. Such a program can recognize the live image of the camera, as well as recognized features or information of the building closure and the drive device and thus offers the possibility of entering commands. This input can be done on the screen by directly clicking on the highlighted information or by menu or buttons. With the help of image recognition algorithms, information that is already contained, e.g. on the main closing edge or on door connection boxes or on different drive types with the functionality of the drive device or settings stored in the drive, is possible. The image of the building closure presented by the camera, as an interaction point, uses a better, individual representation of attachments tailored to the building closure. Attachments can be any recognizable accessory parts of the building lock, such as closing edge protection or gate connection box or light barriers or contacts of a wicket door. By using the AR-enabled device, the commissioning and maintenance of the classic menu navigation via installed buttons or other electrical attachments and countless menus are shown directly on the display of the AR-enabled device and can therefore be used more intuitively.

Image recognition is used on the AR end devices to enable or unlock and thus access the areas that are connected in any way to the building lock viewed via the camera. Humans have the natural ability to recognize what is being seen. The eye takes in the picture, and this picture is compared with memories in the brain. With an image recognition program or corresponding software, only the pixels of an image are recognized and converted into a vector format and compared with known images using an algorithm. This makes it possible for the AR terminal to be available with appropriate programs and a control panel with information output capability and information data processing at the same time. Thus, between the AR terminal and the recognized driving device establish a data connection. But this only if the end device is pointed at the drive device and/or the building closure. Through the camera and the image recognition program, the AR end device recognizes the building closure and the room as such. Such a device in connection with other programs results in different possibilities of use as well as control over and to the building closure and the drive device. The possible uses can be used, for example, in maintenance, servicing, control, documentation and also for everyday use.

For example, parts that need to be checked electrically or mechanically during maintenance work can be displayed and processed in the live image of the camera. Appropriate filters for displaying defects or parts requiring maintenance can also be displayed via the live image with the building closure.

When looking at the interior, the camera, in conjunction with a corresponding program, also recognizes the size of the building closure and the type of structure.

Especially for the maintenance or commissioning area, different actions, such as navigation options through the drive device, are available via the AR terminal for building locks for the operating fitter. A menu control can be selected, for example, in order to monitor the general wear of drive rollers, weight compensation springs, spring break safety devices, e.g. for sectional doors whose positions are changed by lateral carrying cables, it is easy to monitor the carrying cable safety devices. Of course, all sensors or electrical switching elements can be displayed and then specifically checked. This can be experienced as a haptic perception of sensory and/or motor activity. Objects and points of the building closure or the drive mechanism automatically recognized by the programs can provide the viewer with an immediate overview of the building closure by means of image overlays. This is how a program provides important information. This allows menu items, setting options, suggested options and additional filters to be generated and thus the full scope of a menu when setting the drive device to be displayed in the image of the building closure and processed intuitively. By touch operation on the AR-enabled device, on which a program displays the views and images at the same time, the image captures the building closure with the help of the camera and operates the drive device via wireless communication.

The functionality of the AR-enabled device means that the building lock can be commissioned or maintained using an image of the current building lock and without studying the operating instructions for the drive. This increases the convenience and security of commissioning the building lock and also avoids incorrect settings. If errors should nevertheless occur, they can be corrected directly.

In this way, AR can merge the building lock with all stored applications.

In a preferred embodiment, it is also possible for important service intervals to be displayed on the screen of the AR terminal when viewing the drive device, such as the parts that are subject to everyday wear. This enables safer and better quality service, because the technicians can see the drive or drive parts directly in exploded views, for example. At the same time, there is also the possibility of providing the spare parts that are subject to wear with reference numbers and making them visible on the screen.

Work instructions for correcting errors can also be stored and displayed. In particular, a visualization of the assembly of maintenance-intensive parts of the building closure can also be stored and displayed on the screen.

A combination of real and virtual building closure processes is also possible. AR can be used in all areas of the building closure to improve communication via the viewer of the AR terminal.

In order to enable AR applications for building locks, it may be necessary for the real dimensions of the building lock to be recorded beforehand or for the production number to be automatically displayed in the end device. In order to achieve this, the installation situation of the building closure can be clearly determined in both a two- and three-dimensional representation. Due to the constant visual contact, the camera of the end device recognizes the edges of the building closure and then assigns them to the stored information and thus creates a comparison with different levels or levels. This becomes a pattern for the below created for a wide variety of building closures, regardless of the drive side. By aligning the end device, the building closure is recorded with the camera, so to speak. However, the recorded ID can also be used by the data transmission device of the drive device to provide feedback and thus unique identification of the building lock.

AR is an important supporting medium for building locks and leads to an improvement in communication processes through visualization with fast transfer of content, which is a source of information and an activation tool of the viewer.

Collision detection and thus avoidance of accidents with insufficiently maintained building locks is also very important. The AR-capable end device recognizes foreign objects in the area of the building lock using image recognition algorithms stored in a program and can then react to these foreign objects according to one of the command sets stored in a program. A hazard is detected via the camera and in a program-running image recognition algorithm and avoided via a wireless connection to the drive device.

In a preferred embodiment, the AR terminal of the user can be equipped with different levels of service. These service levels can be specially prepared for the service by identifying the user. For example, the service technician can adapt existing applications to company-specific requirements. For this purpose, the actions performed by the service technician are sent to a cloud, for example, in order to document the changes made and make them visible. The cloud represents a form of organization that is operated by the company and is only accessible to the service of the company.

Furthermore, in a further preferred embodiment it is possible that there is also information for the operator of the building lock in special areas on the screen of the AR terminal.

All transmissions of sensor or status data of the entire building closure can be stored directly in the drive device and at the same time in the cloud, with updates being carried out at the same time.

In a special embodiment it is possible that data can also be collected and processed at the same time outside the building lock.

Such an AR terminal can also be used to significantly simplify the installation of a building lock with the connection device, since the position and setting of the building lock can be clearly identified. Likewise, unsafe situations and dangers in the movable area of the building closure can be recognized and corrected at the same time. It is also possible to programmatically control the building lock with visual contact via the AR terminal. With the building lock installation, the building lock operation can be monitored and documented. This makes it possible to set the building closure via the user interface of the end device, because the building closure positions are transmitted live directly to the screen of the AR end device. At the same time, all functions of the building lock can be changed via the screen if required.

Furthermore, simulations for the different types of building closures that are stored on the AR terminal can also be carried out on the AR terminal. The control panel of the AR terminal is used for this. In this respect, the end device represents the interface between the building lock/drive device and a perfect virtual reality via the user interface.

Furthermore, a training program can also be carried out for the service staff using an AR-capable end device, so that they are always up to date with developments.

In the present invention, in contrast to the representation in the prior art, markers are not accessed for tracking. Markers are easily recognizable in particular by shape, contrast, brightness, reflection properties and color, but these are not present or cannot be clearly identified, particularly in older building closures, so that the invention could not be used in such cases with such building closures. For this reason, in particular, the use of markers is dispensed with. Rather, the unmistakable design features of a building lock can be used without a doubt for camera-based tracking.

Due to the existing operability of the AR end device, it is therefore possible to trigger all desired and necessary actions via it. This means that interactions or sys tem states can be changed easily. These can be executed easily and enable, for example, the service personnel to easily navigate through the virtual environment in connection with the information provided.

With a specific program design, it is also possible for service personnel to make changes to the running behavior of the building closure by navigating.

Another way to run mobile applications for automated building locks is to use a web browser as a platform. This can also be done with an integration of programs within the cloud. The cloud offers a considerable degree of flexibility with great application potential for an AR-capable end device.

In addition to a reduction in service costs and less effort in training, the AR terminal enables an interactive process with fast information transfer in the event of type changes or changes to the product, etc.

Different detection signals with clear identification are possible due to the data transmission between the AR terminal device and the drive device. In addition to the serial number, a code or key number can also be used for different Bluetooth IDs for each device. During maintenance, the cycles and the running time of the drive unit with operating hours can be clearly identified as quickly as possible. It is also possible that errors detected in the drive system, including the construction of the building closure, such as running rails and rollers, are reported via the data processing. Maintenance intervals can be carried out safely and with better quality than was previously possible. Necessary spare parts with the reference number can be shown on the display at the same time. Work instructions for correcting errors can also be displayed, for example by visualizing the assembly of both the building closure and the drive device.

With AR, a form of human-technology interaction can be represented in all possible ways in the virtual interaction with the building closure. This means that the digital information merges with the representation of the building closure in the form of gates or doors on site. The gate or door is also recorded through human-computer interaction and provided with additional information for the viewer.

Due to the ever-increasing flood of data, the search time for relevant information is transferred by an AR terminal to a fast switching center with targeted content.

Based on an exemplary representation according to 1 commissioning of a building closure explained in more detail by a flowchart.

A prerequisite for using an AR-enabled terminal device to start up a building lock, for example in the form of a gate or a door, is first of all to establish a connection 1 between the AR-enabled terminal device via a program for starting up the drive device. This connection establishment 1 is carried out via a wireless data connection. According to a camera alignment 2, the camera of the AR-enabled device is then aligned with a view of the entire building closure. After the alignment has been carried out, it is checked whether the building closure is fully recorded and also recognized at the same time. If this is not the case, then a correction is necessary, which requires a new camera alignment 2 . After the dimensions of the building closure have been fully recognized according to a building closure detection 3 , the detection of the building closure 4 is automatically activated in a next program step. In this case, a program unequivocally recognizes the type of building lock and visualizes it on the screen of the AR-enabled device. At the same time, it is possible to use overlays or filters on the screen in order to be taken to other program points or sub-programs for specific actions. In a subsequent menu selection 5, the user can now, for example, select superimpositions or a new display of a menu on symbols that are displayed, and at the same time also change and make settings for the drive device. In a further step, a position determination 6 is activated, which allows the user to mark the opening height and the closed position of the building closure directly on the screen, for example, and also confirm this. This step precisely defines the building closure in terms of its dimensions and placement. The drive device is then reactivated in a learning drive 7 for locking the building. This activation is carried out via the screen in a quasi "dead man's switch". The building lock moves from its closed position to the defined open position. During this learning run, the building closure is constantly monitored by the camera. Should there be any irregularities here, a carried out a new learning run. If the program shows the sequence of the learning drive as correct in connection with the visual check, a further program point, namely route free 8, is activated. In this program item, it is checked again whether the building closure is still completely in the field of view of the camera. If this is not the case, the drive device is shut down according to program point drive stop 9 and a new check is carried out.

A general drive release 10 is activated if the program item Clear route 8 is checked and reproduced correctly. In a further program item Position assumption 11, the program can be checked again for security, whether all positions have been learned and can also be reached by the building lock. If this is not the case, there will be a new reference to the program point Route free 8 and the previous program point will be carried out again in the procedure. If, however, all required positions are correctly checked with a program or a program point and found to be correct, the program point of position confirmation 12 proceeds. Here the user of the AR-enabled device is asked to confirm the positions and at the same time can activate another section of security for the building lock. This point is that after all positions have been correctly confirmed, it is then necessary that the forces occurring at the leading closing edge are taught to the maximum allowable magnitude. For this purpose, the program step Track clear 8 is activated again and checked again whether the camera is still capturing the entire building closure. If this is not the case, a repeated check is activated via the drive stop 9, which must then be carried out manually. If, however, the correct camera alignment is confirmed, the program point force learning run ended 14 is activated by the drive enable 10 . If, however, the force learning drive was not carried out correctly, which is checked internally within the AR-capable device via stored information about the building closure, a return is made to the program item Route free 8 in order to carry out a correction in a renewed execution of the force learning drive. If, however, the information from the force-learning trip is complete 14 correct, the commissioning 13 of the building lock can be completed.

This preceding description of starting up a building lock is only to be seen as an example, which means that depending on the type of building lock and in particular the drive device, different sequences in different programs or program steps are possible here.

Reference List

1

connection establishment

2

camera orientation

3

building closure detection

4

Program step for building closure detection

5

menu selection

6

positioning

7

building learning trip

8th

road clear

9

drive stop

10

drive enable

11

position taking

12

position confirmation

13

Installation

14

Force learning run ended

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• US66225299B1 [0002]
• DE 102015215613 A1 [0003]

Claims (17)

Device for diagnostics and/or commissioning and/or assembly of building locks, which are equipped with a drive device for opening and closing, the device consisting of a mobile augmented reality (AR) - capable end device, which in addition to a real image of the building lock can also display at least one virtual image or image components of the building closure with the drive device on a screen, for which purpose the AR terminal has a camera for actually recording an existing building closure in or on a building, and a storage medium for storing documentation data on different building closure designs and drive devices that are stored in different programs within the AR terminal and can be executed by a data processing unit with wireless data transmission between the drive device of the real building closure and the mobile AR terminal with the processing of the real image components recorded by the camera, a control panel for communication with the drive device of the real building lock, and a data connection device between the drive device of the real building lock and the AR terminal. device after claim 1 , characterized in that the wireless data transmission between the drive device of the real building lock and the mobile terminal device can be carried out via a Bluetooth connection or WLAN or a smart home system. device after claim 1 , characterized in that the terminal is equipped with different service levels for different services, the service levels being stored in the programs. Device according to one or more of the preceding claims, characterized in that the stored programs can be updated automatically after the actual recording of the building closure by the camera, with the program update being executable either by information stored in a cloud or in the drive device. device after claim 1 , characterized in that the virtual image or the image components can have current or stored information about the real building closure and/or the drive device for diagnostics and/or maintenance and/or installation information. device after claim 5 , characterized in that the stored and / or current information of the real building lock and / or the drive device have maintenance information and / or usage information. device after claim 6 , characterized in that the maintenance information can include status information about safety-relevant parts and / or service-prone facilities of the building lock and the drive device. device after claim 6 , characterized in that further information on the installation of additional equipment of the building closure can be transmitted by the assembly of the building closure in a building niche the installer after the alignment of the camera on the real building closure. device after claim 6 , characterized in that after the installation of the building lock the fitter can call up the information for commissioning the building lock on the AR terminal step by step by aligning the camera on the building lock. device after claim 1 , characterized in that the data connection device is integrated into the drive device or can be subsequently connected to the drive device. device after claim 1 , characterized in that the AR-enabled terminal is a smartphone or tablet. Method for using an augmented reality (AR) - capable, mobile terminal for an object-oriented use on a building closure with a drive device and an alignment of the AR terminal on the building closure, so that the user on a screen via the camera present in the terminal the building closure is displayed as a real object, with documentation data being transmitted simultaneously with immersive information via a wireless data connection device in both directions between the AR terminal and the relevant driving device of the building closure, so that these can be displayed separately or simultaneously on the AR- Terminal can be displayed. procedure after claim 12 , characterized in that the AR terminal has storage means for storing documentation of different building designs and their drive devices. Procedure according to one of Claims 12 and 13 , characterized in that the AR terminal can access the processing of stored documentation data from the drive device via a data processing unit using a Bluetooth connection or W-LAN or a smart home system and can be processed with current information on the real building lock and the drive device. procedure after Claim 13 , characterized in that the information of the building closure and / or the drive device is transmitted as maintenance information and / or usage information, the information on the AR terminal can be called up step by step. procedure after claim 12 , characterized in that the mobile AR terminal is provided with a control panel via which information about the drive device is transmitted by means of wireless data transmission when there is visual contact with the real building closure. Method according to one or more of the preceding Claims 12 until 16 , characterized in that the stored programs are automatically updated after the camera has taken the picture of the building closure, with the program updates being carried out either by information stored in the cloud or in the drive device.

Images