WO2014130417A1

WO2014130417A1 - Multi disciplinary engineering design using image recognition

Info

Publication number: WO2014130417A1
Application number: PCT/US2014/016799
Authority: WO
Inventors: Oswin Noetzelmann; Rami Reuveni; Victor Robert HAMBRIDGE; Marine DUREL; Tim OERTER; JR. Christopher Patrick PORTWAY; Dirk VIELSAECKER; Sarvananthan RAGAVAN; Mingjun Zhang; Daniela Stederoth
Original assignee: Siemens Aktiengesellschaft; Siemens Product Lifecycle Management Software Inc.
Priority date: 2013-02-19
Filing date: 2014-02-18
Publication date: 2014-08-28

Abstract

A method of using image recognition for cross domain data exchange includes receiving (4-1) a set of paired graphical objects, wherein a first graphical object is read from an online library, and a second graphical object is an external graphics object, transferring (4-2) tags and classification associated with the first graphical object onto one or more other external representations of said first graphical object, and presenting (4-3) a user with a collection of external graphical objects that match the tag and classification criteria transferred from the first graphical object. The online library may be updated (4-4) with one or more of said matched external graphical objects, if one or more of said matched external graphical objects are not already contained in said library

Description

MULTI DISCIPLINARY ENGINEERING DESIGN USING IMAGE RECOGNITION

Cross Reference to Related United States Applications

This application claims priority from "Method for Engineering Design using Image Recognition in a Multidisciplinary Engineering System", U.S. Provisional Application No. 61/766,160 of Noetzelmann, et al., filed February 19, 2013, the contents of all of which are herein incorporated by reference in their entireties.

Technical Field

This disclosure is directed to methods for cross-domain data exchange.

Discussion of the Related Art

Communication between disciplines is often still through exchanging primitive graphics, such as schematics, which the receiving discipline will recognize as discipline specific objects or object categories. The receiving parry will use this information to build their own discipline-specific representation. However, this process can be assisted, or even automated completely, through using image recognition (IR) and optical character recognition (OCR) techniques.

Image recognition is commonly tailored to recognizing specific types of objects, such as faces in security software, books and other media in cell phone shopping applications, or text in the case of Optical Character Recognition (OCR). This specialization is traditionally built into the specific image recognition system, or added as a module created a priori.

Some recent approaches combine image type specialization with after- deployment user-added tagging. One example is automatic tagging in online social network systems, where a user can tag a face with a name, and subsequently future images will automatically apply the tag where the face is recognized. Google has also combined automatic tagging with user input in Google Image Labeler, where the user directly inputs tags associated with image content. Traditional engineering disciplines work is separated from a data point of view, and manual synchronization of discipline specific data is time consuming and error prone. A typical synchronization workflow may include importing data, and selecting a library object to replace an imported object based on the imported object's image and possibly tags and classification. The user may rely on the visual representation of the imported external object and the library object to make this match.

FIG. 1 depicts external objects as being represented by outlined shapes, and the parameterized data as solid filled shapes. In FIG. 2, the user manually matches the external and parameterized objects, one at a time.

A multi-disciplinary engineering system is a system that allows engineers from multiple disciplines to work on common or connected data. For example, as illustrated in FIG. 3, a factory planner can work together with management, a mechanical engineer, an electrical engineer and an automation engineer to plan a new production line for a car door assembly.

Traditionally, the disciplines work is separated from a data point of view and manual synchronization of the discipline specific data is time consuming and error prone. For example, when the automation engineer introduces a new programmable logic controller to the project, which he needs to automate the line, this information needs to be transported to the electrical engineer, so he can place it in the right electrical cabinet and plan its wiring. If this information is not transmitted or distorted, it can have serious impact on the quality of the work of the electrical engineer and vice versa.

Data connections between the different disciplines can potentially be used to allow the system to support various functions: (1) Notification/communication between disciplines or departments; (2) Rule based change propagation; (3) Formalization of workflows, such as sign off procedures; (4) Multi-Disciplinary report generation; and (5) Usage of common interdisciplinary data structures. Summary

Exemplary embodiments of the invention as described herein generally include methods for cross domain data exchange, through matching non- parameterized graphics and parameterized data which also contain graphic representations. Exemplary embodiments of the invention use image recognition (IR) and optical character recognition (OCR) techniques to build multi-disciplinary objects from domain specific objects and to update a shared library. Embodiments of the invention can be implemented as an integrated layer or as an add-in, within a multidisciplinary tool or in domain specific tools.

According to an aspect of the invention, there is provided a computer- implemented method of using image recognition for cross domain data exchange, including receiving a set of paired graphical objects, wherein a first graphical object is read from an online library, and a second graphical object is an external graphics object, transferring tags and classification associated with the first graphical object onto one or more other external representations of said irst graphical object, and presenting a user with a collection of external graphical objects that match the tag and classification criteria transferred from the first graphical object.

According to a further aspect of the invention, the method includes updating said online library with one or more of said matched external graphical objects, if one or more of said matched external graphical objects are not already contained in said library.

According to a further aspect of the invention, the method includes updating said online library with an encapsulation of one or more of said matched external graphical objects and the first graphical object within one of a new graphical object in the library, or the original first graphical object as one or more additional representations.

According to a further aspect of the invention, the set of paired graphical objects were paired by a user. According to a further aspect of the invention, the set of paired graphical objects were automatically paired by an image recognition program.

According to a further aspect of the invention, the image recognition program does not use pre-defined rules or user tags to pair said set of paired graphical objects.

According to a further aspect of the invention, pairing said set of paired graphical objects comprises matching a projected image of a 3-dimensional representation of the first graphical object with a 2-dimensional representation of said second graphical object.

According to a further aspect of the invention, the method includes receiving thresholds for recognizing similar objects, and tolerances for image recognition facets.

According to another aspect of the invention, there is provided a computer- implemented method of using image recognition for cross domain data exchange, including importing a 2-dimensional (2D) layout image, recognizing and cataloging objects in said 2D layout image, and collecting text associated with said objects in said layout image, receiving representations of one or more 3-dimensional (3D) objects placed on top of one or more recognized objects in said 2D layout image, wherein said 3D objects were extracted from an online library, extracting classifications associated with said placed 3D objects and applying said classifications to objects in said 2D image that are covered by the 3D objects, and adding the covered object with the applied classifications to the online library.

According to another aspect of the invention, there is provided a non- transitory program storage device readable by a computer, tangibly embodying a program of instructions executed by the computer to perform the method steps for using image recognition for cross domain data exchange

Brief Description of the Drawings

FIG. 1 illustrates the importation of data and selection of a library object to replace an imported object, according to an embodiment of the invention. FIG. 2 illustrates a user manually matching the external and parameterized objects, one at a time, according to an embodiment of the invention.

FIG. 3 illustrates how engineers from multiple disciplines can work on common or connected data, according to an embodiment of the invention.

FIG. 4 depicts a workflow according to an embodiment of the invention.

FIG. 5 depicts another workflow according to an embodiment of the invention.

FIG. 6 is a block diagram of an exemplary computer system for implementing a method for using image recognition in a multidisciplinary engineering system, according to an embodiment of the invention.

Detailed Description of Exemplary Embodiments

Exemplary embodiments of the invention as described herein generally include systems and methods for using image recognition in a multidisciplinary engineering system. Accordingly, while the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that there is no intent to limit the invention to the particular forms disclosed, but on the contrary, the invention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

As used herein, the term "image" refers to multi-dimensional data composed of discrete image elements (e.g., pixels for 2-dimensional images and voxels for 3- dimensional images). The image may be, for example, a factory computer-aided design (CAD) layout, which may be a 2-dimensional or 3-dimensional pixel image or a 2 or 3-dimensional vector source image. Although an image can be thought of as a function from R² or R³ to R or R⁷, the methods of the inventions are not limited to such images, and can be applied to images of any dimension, e.g., a 2- dimensional picture or a 3- dimensional volume. For a 2- or 3-dimensional pixel image, the domain of the image is typically a 2- or 3-dimensional rectangular array, wherein each pixel or voxel can be addressed with reference to a set of 2 or 3 mutually orthogonal axes. The terms "digital" and "digitized" as used herein will refer to images or volumes, as appropriate, in a digital or digitized format acquired via a digital acquisition system or via conversion from an analog image.

In a workflow according to an embodiment of the invention, it is assumed that an external graphical input is present, and that there is a library of reusable graphical objects. This library should be multi-disciplinary in nature, potentially holding multiple visual representations and other attributes, including tags and classifications. Each representation and attribute can be marked as belonging to one or more disciplines or shared by all.

In a workflow according to an embodiment of the invention, depicted in FIG. 4, a user may pair the visual representation of a graphical library object with another visual representation from an external source, herein referred to as an external representation, and an IR system according to an embodiment of the invention provides assistance in pairing related internal objects. In FIG. 4, the box labeled "New Module" between the "Library" box and the "Graphics" box represents an integrated layer or an add-on according to an embodiment of the disclosure.

Referring now to FIG. 4, a workflow may begin at step 4-1 with a user pairing a graphical object from the Library to an external graphics object. Once this pairing is performed for one external representation, any other external representations identified as the same will inherit some or all of the tags and classifications from the library object at step 4-2. At step 4-3, the user may then receive a visual indication that an additional action is available, and may be presented with a pruned collection of external objects that match tag and classification criteria learned from the pairing, from which the user may chose another external representation. The external representation may be in a domain not already contained in the library object. In that case, if desired, the library object can be updated at step 4-4 by adding the external representation as a domain-specific representation.

In some domains, the pairing of library objects to imported external representations is a common user workflow. However, embodiments of the invention can allow a user to perform their normal workflow with existing tools, and at any point switch to and from an accelerated workflow. Embodiments of the invention do not require pre-defined rules or user tags or classification input for the mapping. In addition, embodiments of the invention can update a library object to contain an additional discipline context, without any change in the user's prior workflow. The resulting object can be used directly in other disciplines if desired, thus encouraging multidisciplinary collaboration with little overhead.

While each discipline can have its own internal representation, the underlying object that is being matched is often the same real-world, purchasable item. For this reason, according to embodiments of the invention, the matching can sometimes be entirely automated. In a case of a 2-dimentional image as an external representation, a simplified projected image of a 3-dimentional representation in a library object may be matched to the 2-dimensional input without any explicit user-matching. This automatic matching can reduce overhead for multi-disciplinary collaboration, since some objects can gain data from additional disciplines with little to no user interaction. A 3 dimensional object model can be paired with a 2-dimensional projection without using any manual or implied pairing information, such as might be acquired from a training session. A user can also enter additional details before matching for more exact results, while still spending less time than with current manual or assisted manual pairing methods.

According to embodiments of the invention, a user may change additional settings to improve options presented during assisted manual pairing and automatic pairing results. Some examples are as follows: (1) Thresholds for recognizing similar images, i.e. two objects in an external image with a same figure, but different widths, could be considered the same object. Similar settings can be made for rotation. (2) Tolerances for image recognition facets, i.e. rungs on a ladder or doors in an electrical cabinet may vary within the same classifications. In some cases, the classifications may change based on user-editable ranges for the tolerances.

An exemplary workflow according to an embodiment of the invention is illustrated in FIG. 5. A workflow according to an embodiment of the disclosure may begin at step 5-1 by a line design engineer importing a "carpet," a 2 dimensional sketch of a plant layout in a top view format. Graphical objects or sub-images in the carpet may be delineated and cataloged by an image recognition system according to an embodiment of the invention, and any nearby text may be collected to be associated with these sub-images. At step 5-2, this engineer places 3 dimensional objects, possibly specific to the line design domain, on top of individual or compound sub-images. At step 5-3, the existing classification may be extracted from the placed object and applied to the covered image, which has been already been delineated and cataloged by an image recognition system according to an embodiment of the invention. The covered image may be added as an additional representation for the library object. If there are more instances of the now classified 2 dimensional image on the same carpet: a user may now click on it at step 5-4a to filter the library, or bring up a dialog box to do the same. The user may be provided with an option to automatically place a new instance of the last placed part on a similarly cataloged part, to select from the library within the classification or within ancestor- classifications, or to place a new item on the last placed part. At step 5-4b, when the new item is placed, the classification is now appended to the existing classifications of the 2 dimensional image, if it is different from previous classifications. Steps 5-4a-b may be repeated, depending on user settings, future carpets in the project, in that office, or in that company, thus skipping the manual 3 dimensional object search of step 52. Useful suggestions can be quickly reached when the user or an administrative user manually adds, edits, or deletes 2 dimensional object-to- classification mappings, or applies a mapping file to perform these operations en masse.

Utilizing a matching framework according to an embodiment of the disclosure, engineers can save much of the effort of synchronizing data between disciplines through object pairing. An automatic process according to an embodiment of the disclosure may replace a manual process that can take significant engineering time to match object domain representations. Different engineering domains may finish their work earlier. For example, a plant engineering project can be finished earlier, which means that the plant can start production earlier. In addition, an automatic process is less error prone than an entirely manual pairing step. Depending on project and user preferences, methods according to embodiments of the disclosure can augment manual pairing and synchronization, or completely automate it once sufficient manual pairing has been performed, or an existing pairing database has been loaded. It is possible to adapt a method according to embodiments of the disclosure to all existing and future multi-disciplinary engineering systems. It is to be understood that embodiments of the present disclosure can be implemented in various forms of hardware, software, firmware, special purpose processes, or a combination thereof. In one embodiment, the present invention can be implemented in software as an application program tangible embodied on a computer readable program storage device. The application program can be uploaded to, and executed by, a machine comprising any suitable architecture.

FIG. 6 is a block diagram of an exemplary computer system for implementing a method for using image recognition in a multidisciplinary engineering system, according to an embodiment of the invention. Referring now to FIG. 6, a computer system 61 for implementing the present invention can comprise, inter alia, a central processing unit (CPU) 62, a memory 63 and an input/output (I/O) interface 64. The computer system 61 is generally coupled through the I/O interface 64 to a display 65 and various input devices 66 such as a mouse and a keyboard. The support circuits can include circuits such as cache, power supplies, clock circuits, and a communication bus. The memory 63 can include random access memory (RAM), read only memory (ROM), disk drive, tape drive, etc., or a combinations thereof. The present invention can be implemented as a routine 67 that is stored in memory 63 and executed by the CPU 62 to process the signal from the signal source 68. As such, the computer system 61 is a general purpose computer system that becomes a specific purpose computer system when executing the routine 67 of the present invention.

The computer system 61 also includes an operating system and micro instruction code. The various processes and functions described herein can either be part of the micro instruction code or part of the application program (or combination thereof) which is executed via the operating system. In addition, various other peripheral devices can be connected to the computer platform such as an additional data storage device and a printing device.

It is to be further understood that, because some of the constituent system components and method steps depicted in the accompanying figures can be implemented in software, the actual connections between the systems components (or the process steps) may differ depending upon the manner in which the present invention is programmed. Given the teachings of the present invention provided herein, one of ordinary skill in the related art will be able to contemplate these and similar implementations or configurations of the present invention.

While the present invention has been described in detail with reference to exemplary embodiments, those skilled in the art will appreciate that various modifications and substitutions can be made thereto without departing from the spirit and scope of the invention as set forth in the appended claims.

Claims

CLAIMS What is claimed is:

1. A computer-implemented method of using image recognition for cross domain data exchange, the method executed by the computer comprising the steps of: receiving a set of paired graphical objects, wherein a first graphical object is read from an online library, and a second graphical object is an external graphics object;

transferring tags and classification associated with the first graphical object onto one or more other external representations of said first graphical object; and presenting a user with a collection of external graphical objects that match the tag and classification criteria transferred from the first graphical object.

2. The method of claim 1, further comprising updating said online library with one or more of said matched external graphical objects, if one or more of said matched external graphical objects are not already contained in said library.

3. The method of claim 1 , further comprising updating said online library with an encapsulation of one or more of said matched external graphical objects and the first graphical object within one of a new graphical object in the library, or the original first graphical object as one or more additional representations.

4. The method of claim 1 , wherein the set of paired graphical objects were paired by a user.

5. The method of claim 1 , wherein the set of paired graphical objects were automatically paired by an image recognition program.

6. The method of claim 5, wherein said image recognition program does not use pre-defined rules or user tags to pair said set of paired graphical objects.

7. The method of claim 5, wherein pairing said set of paired graphical objects comprises matching a projected image of a 3-dimensional representation of the first graphical object with a 2-dimensional representation of said second graphical object.

8. The method of claim 1 , further comprising receiving thresholds for recognizing similar objects, and tolerances for image recognition facets.

9. A computer-implemented method of using image recognition for cross domain data exchange, the method executed by the computer comprising the steps of: importing a 2-dimensional (2D) layout image;

recognizing and cataloging objects in said 2D layout image, and collecting text associated with said objects in said layout image;

receiving representations of one or more 3-dimensional (3D) objects placed on top of one or more recognized objects in said 2D layout image, wherein said 3D objects were extracted from an online library;

extracting classifications associated with said placed 3D objects and applying said classifications to objects in said 2D image that are covered by the 3D objects; and adding the covered object with the applied classifications to the online library.

10. A non-transitory program storage device readable by a computer, tangibly embodying a program of instructions executed by the computer to perform the method steps for using image recognition for cross domain data exchange, the method comprising the steps of:

receiving a set of paired graphical objects, wherein a first graphical object is read from an online library, and a second graphical object is an external graphics object;

1 1. The computer readable program storage device of claim 10, the method further comprising updating said online library with one or more of said matched external graphical objects, if one or more of said matched external graphical objects are not already contained in said library.

12. The computer readable program storage device of claim 10, the method further comprising updating said online library with an encapsulation of one or more of said matched external graphical objects and the first graphical object within one of a new graphical object in the library, or the original first graphical object as one or more additional representations.

13. The computer readable program storage device of claim 10, wherein the set of paired graphical objects were paired by a user.

14. The computer readable program storage device of claim 10, wherein the set of paired graphical objects were automatically paired by an image recognition program.

15. The computer readable program storage device of claim 14, wherein said image recognition program does not use pre-defined rules or user tags to pair said set of paired graphical objects.

16. The computer readable program storage device of claim 14, wherein pairing said set of paired graphical objects comprises matching a projected image of a 3-dimensional representation of the first graphical object with a 2-dimensional representation of said second graphical object.

17. The computer readable program storage device of claim 10, the method further comprising receiving thresholds for recognizing similar objects, and tolerances for image recognition facets.