WO2023075308A1

WO2023075308A1 - Electronic device for guiding maintenance of product and method for operating electronic device

Info

Publication number: WO2023075308A1
Application number: PCT/KR2022/016142
Authority: WO
Inventors: 유철원
Original assignee: (주)메가플랜
Priority date: 2021-10-26
Filing date: 2022-10-21
Publication date: 2023-05-04

Abstract

Provided is a method for operating an electronic device, the method comprising the steps of: capturing an image of a product; on the basis of the result of identifying a plurality of elements in the image, displaying status information representing maintenance statuses of the plurality of elements; on the basis of the status information, receiving an input of selecting a first element from the plurality of elements; and displaying first guide information for guiding maintenance of the first element.

Description

Electronic devices that guide maintenance of the product and how they operate

It relates to an electronic device for guiding maintenance of a product and an operating method of the electronic device.

Recently, the development and demand for content is rapidly increasing, and manuals for products are also produced using content authoring tools such as Unity3D. In particular, since manuals using augmented reality are not accurately matched between information and manuals that actual users want to use, it is difficult for users to use the manuals.

Therefore, a technique for enabling users to easily use manuals for products and providing accurate information is required.

When an image of a product is obtained, information indicating the maintenance status of the product is displayed on the image and guide information guiding maintenance of the selected element is displayed, thereby guiding the maintenance information and maintenance of the product on the image of the product. We want to provide a variety of information by augmenting information.

According to one side, obtaining an image for the product; based on a result of identifying a plurality of elements in the image, displaying status information indicating a maintenance state of the plurality of elements; receiving an input for selecting a first element from among the plurality of elements based on the state information; and displaying first guide information for guiding maintenance of the first element.

Various types of information may be provided by augmenting product state information and guide information for guiding maintenance on a product image.

It is possible to improve the understanding of mechanics by increasing the degree of matching between information required for product maintenance and manual information.

BRIEF DESCRIPTION OF THE DRAWINGS The present invention may be readily understood from the following detailed description in combination with the accompanying drawings, wherein reference numerals denote structural elements.

1 is a diagram of an electronic device displaying status information indicating a maintenance state of elements identified in an image and displaying guide information for guiding maintenance of a selected element, according to an embodiment, when an image of a product is obtained. It is a conceptual diagram to explain the operation.

2 is a flowchart illustrating a method of operating an electronic device according to an exemplary embodiment.

3 is a diagram for explaining a result of identifying a plurality of elements in an image as an image of a product is obtained according to an embodiment.

4A and 4B are diagrams for explaining a screen displaying status information indicating a maintenance status of a plurality of elements, according to an exemplary embodiment.

5 is a diagram for explaining a screen displaying guide information for guiding element maintenance, according to an exemplary embodiment.

6 is a diagram for explaining a screen displaying guide information according to an exemplary embodiment.

7 is a diagram for explaining a screen displayed according to a maintenance progress of a product according to an embodiment.

8 is a block diagram illustrating a configuration of an electronic device according to an exemplary embodiment.

Hereinafter, various embodiments will be described in detail with reference to the drawings. Embodiments described below may be modified and implemented in various different forms. In order to more clearly describe the characteristics of the embodiments, a detailed description of matters widely known to those skilled in the art to which the following embodiments belong will be omitted.

On the other hand, when a component is said to be "connected" to another component in this specification, this includes not only the case of being 'directly connected', but also the case of being 'connected with another component in between'. In addition, when a certain component "includes" another component, this means that other components may be further included without excluding other components unless otherwise specified.

Also, terms including ordinal numbers such as 'first' or 'second' used in this specification may be used to describe various elements, but the elements should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another.

1 is an electronic device displaying status information indicating the maintenance status of elements identified in the image and displaying guide information for guiding the maintenance of a selected element according to an embodiment, when an image of a product is acquired ( It is a conceptual diagram to explain the operation of 10).

Referring to the image 110 of FIG. 1 , the electronic device 10 may obtain an image of a product through a camera in the electronic device 10 or an external camera connected to the electronic device 10 . For example, the electronic device 10 may acquire an image of an engine room of a vehicle.

The electronic device 10 may identify a plurality of elements constituting the engine room on the image and display name information of the plurality of elements. Also, the electronic device 10 may display status information on an element requiring maintenance among a plurality of elements. For example, the electronic device 10 may display

icons

421, 422, and 423 indicating state information of engine oil, washer fluid, and battery among a plurality of elements. When an input for selecting the icon 421 representing the state of engine oil is received from among the

icons

421, 322, and 423, the electronic device 10, as shown in the image 120 of FIG. 1, selects engine oil. A video 510 guiding a replacement method and information 520 guiding an engine oil replacement tip may be displayed. When an input for selecting the icon 511 for playing the video 510 is received, the electronic device 10 can play the video 510.

2 is a flowchart illustrating an operating method of the electronic device 10 according to an exemplary embodiment.

Referring to FIG. 2 , in step S210, the electronic device 10 may acquire an image of the product. For example, a product may refer to an object requiring work such as assembly and maintenance.

For example, the electronic device 10 may obtain an image of a product through a camera in an electric car device. For another example, the electronic device 10 may acquire an image of a product through an external camera connected to the electronic device 10 . Here, the external camera may capture maintenance progress of the product in real time and transmit the captured image to the electronic device 10 .

For example, the electronic device 10 may acquire at least one image of a product based on a level representing a maintenance progress state of the product. For example, a level indicating a maintenance progress state may be classified according to a maintenance completion rate.

In operation S220 , the electronic device 10 may display status information indicating a maintenance status of a plurality of elements based on a result of identifying a plurality of elements in the image.

For example, the electronic device 10 may detect a plurality of objects in an image. The electronic device 10 may identify a plurality of elements corresponding to a plurality of objects based on shape information or label information of each of the plurality of objects. The electronic device 10 may obtain state information indicating a maintenance state of a plurality of elements based on at least one of an assembly relationship between a plurality of elements, external information of the plurality of elements, and use history information of the plurality of elements. there is.

For example, the electronic device 10 may determine the priority of at least one element requiring maintenance among a plurality of elements based on state information of the plurality of elements. The electronic device 10 may display state information of at least one element based on the priority of the at least one element.

For example, the electronic device 10 may display status information including at least one of a name of each of a plurality of elements, appearance information, and use history information.

In operation S230, the electronic device 10 may receive an input for selecting a first element from among a plurality of elements based on state information.

For example, the electronic device 10 may classify and display elements that require maintenance, elements that have completed maintenance, and elements that do not require maintenance on an image. For example, the electronic device 10 may display elements according to a display method corresponding to elements requiring maintenance, elements for which maintenance has been completed, and elements not requiring maintenance. For example, the display method may include a method of differently displaying colors for each type. For example, the electronic device 10 may receive an input for selecting a first element among elements requiring maintenance on an image. For another example, the electronic device 10 may receive an input for selecting an icon representing state information of the first element.

In step S240, the electronic device 10 may display first guide information for guiding maintenance of the first element.

For example, the electronic device 10 may display first guide information including at least one of a name of the first element, a replacement period, and a maintenance method.

For example, when maintenance of the first element is completed, the electronic device 10 performs maintenance of a second element requiring subsequent maintenance work after the maintenance work of the first element, based on state information of a plurality of elements. Second guide information for guiding may be displayed.

For example, the electronic device 10 may display state information of a plurality of elements and first guide information of a first element on one screen on an image.

For example, the electronic device 10 may display information about a maintenance progress status and expected maintenance time according to the maintenance progress of the first element.

For example, the electronic device 10 may display a first execution icon for accessing schedule information of a mechanic who maintains a product. Also, the electronic device 10 may display a second execution icon for accessing contact information of a customer who has requested maintenance of the product.

The electronic device 10 may acquire an image of a product through a camera within the electronic device 10 or an external camera connected to the electronic device 10 . As shown in FIG. 3 , the electronic device 10 may acquire an image of an engine room of a vehicle.

When an image is obtained, the electronic device 10 may detect a plurality of objects in the image. The electronic device 10 may identify a plurality of elements corresponding to a plurality of objects based on shape information or label information of each of the plurality of objects.

Referring to FIG. 3 , the electronic device 10 may detect a plurality of objects constituting an engine room of a vehicle in an image. For example, a plurality of objects may be detected based on a learning model that detects elements constituting an engine room from an image obtained by capturing an engine room of a vehicle. For example, the learning model may be obtained by learning external information or label information of an object in an image obtained by photographing a plurality of engine rooms.

The electronic device 10 includes an engine 301, an engine oil inlet 302, an engine oil level gauge 303, a coolant auxiliary tank 304, a washer fluid tank 305, Each of the brake oil tank 306 , the air cleaner 307 , the transmission oil level gauge 308 , the fuse box 309 , and the battery 310 can be detected. On the image, the electronic device 10 includes an engine 301, an engine oil inlet 302, an engine oil level gauge 303, a coolant auxiliary tank 304, a washer fluid tank 305, and a brake oil tank 306 Name information for each of the air cleaner 307, the transmission oil level gauge 308, the fuse box 309, and the battery 310 may be displayed. Also, the electronic device 10 may display number information for each of a plurality of elements constituting the engine room.

The electronic device 10 may determine the priority of at least one element requiring maintenance from among the plurality of elements, based on state information of the plurality of elements. Here, the priority may be set higher as the replacement time of the element approaches. For example, the replacement timing may be determined based on a vehicle driving distance or vehicle driving time that has elapsed since the previous replacement timing.

Referring to FIG. 4A , the electronic device 10 may determine that engine oil, coolant, brake oil, air cleaner, transmission oil, and battery should be replaced based on state information of a plurality of elements. The electronic device 10 may set a higher priority for an element requiring replacement as the replacement time approaches. For example, if the element replacement period is within 2 weeks, it is set as 1st priority, if the element replacement period is within 8 weeks, it is set as 2nd priority, and if the element replacement period is within 12 weeks, it is set as 3rd priority. It can be. Since the priority is determined according to the replacement time, there may be elements having the same priority. For example, based on the state information of the elements, the electronic device 10 gives engine oil first priority, coolant priority 3 priority, brake oil priority 2, air cleaner priority 2, transmission oil priority 3, and battery priority. It can be set to 3 priorities.

The electronic device 10 may display information corresponding to the priority based on the priority of elements requiring replacement. For example, the electronic device 10 may display three asterisks for a first-rank element, two asterisks for a second-rank element, and one asterisk for a third-rank element. As shown in FIG. 4A , the electronic device 10 includes information 401 indicating that it is the first priority for engine oil, information indicating that it is the third priority for cooling water 402, and information indicating that it is the second priority for brake oil ( 403), information 404 indicating the second priority for the air cleaner, information indicating the third priority for the mission oil 405, and information 406 indicating the third priority for the battery may be displayed.

The electronic device 10 may obtain state information indicating a maintenance state of a plurality of elements based on at least one of an assembly relationship between a plurality of elements, external information of the plurality of elements, and use history information of the plurality of elements. there is. For example, the electronic device 10 may display status information about an element for which maintenance is requested among a plurality of elements constituting the engine room. For example, the electronic device 10 may display an icon representing state information of an element.

For example, as shown in FIG. 4B , the electronic device 10 may display

icons

421 , 422 , and 423 indicating state information of engine oil, washer fluid, and battery among a plurality of elements. By displaying the

icons

421, 422, and 423 on the image, the mechanic can recognize that engine oil replacement, washer fluid replenishment, and battery replacement are necessary. For example, the order in which

icons

421, 422, and 423 are displayed may be determined based on replacement time.

For example, the electronic device 10 may receive an input for selecting a region corresponding to engine oil on the image. For example, the electronic device 10 may receive an input for selecting a region including engine oil or an icon representing a state of engine oil on an image. The electronic device 10 may display guide information guiding replacement of engine oil.

For example, as shown in FIG. 5 , the electronic device 10 may display a video 510 guiding how to replace engine oil and information 520 guiding engine oil replacement tips. When an input for selecting the icon 511 for playing the video 510 is received, the electronic device 10 can play the video 510 guiding how to replace engine oil.

Referring to FIG. 6 , the electronic device 10 may display state information of a plurality of elements and first guide information of a first element on one screen on an image. The electronic device 10 may display information on a plurality of elements and state information on an element requiring maintenance among the plurality of elements on the image of the engine room, on the first area 610 of the screen. Also, the electronic device 10 may display guide information for guiding maintenance of the selected element on the second area 620 of the screen.

By displaying state information of a plurality of elements and guide information for guiding maintenance of a selected element on one screen, a mechanic can grasp the maintenance status of a plurality of elements and manual information on the selected element on one screen.

The electronic device 10 may display information on a maintenance progress status and expected maintenance time according to the maintenance progress of the product. For example, as described with reference to FIG. 4B , the electronic device 10 may display an icon representing state information of an element for which maintenance is requested among a plurality of elements constituting the engine room. The electronic device 10 may display icons indicating state information of engine oil, washer fluid, and battery among a plurality of elements. For example, when the replacement of engine oil is completed, an icon indicating state information of engine oil may be removed from the image. Also, when the replenishment of the washer fluid is completed, an icon representing state information of the washer fluid may be removed from the image. If the battery is not replaced, as shown in FIG. 7 , the icon 423 indicating battery state information may be displayed without being removed.

In this case, the electronic device 10 may display maintenance progress information 711 indicating a state in which maintenance of three elements, among three elements requiring maintenance work, has been completed. In addition, the electronic device 10 may display estimated maintenance time information 712 required for battery replacement.

Meanwhile, the electronic device 10 may display a first execution icon 713 for accessing schedule information of a mechanic. When an input for selecting the first execution icon 713 is received, the electronic device 10 may execute the mechanic's schedule application and display registered schedule information of the mechanic. For example, schedule information may include maintenance reservation information.

Also, the electronic device 10 may display a second execution icon 714 for accessing contact information of a customer who has requested maintenance. When an input for selecting the second execution icon 714 is received, the electronic device 10 may execute the phone application and attempt to make a phone call to the customer.

By displaying the first execution icon 713 and the second execution icon 714, the technician can efficiently manage schedules and quickly contact customers of products under maintenance.

8 is a block diagram illustrating a configuration of an electronic device 10 according to an exemplary embodiment.

Referring to FIG. 8 , the electronic device 10 may include a communication unit 810, a camera 820, an output unit 830, a memory 840, and a processor 850. The electronic device 10 may further include other components in addition to the components shown in FIG. 8 . Also, the electronic device 10 may include fewer components than those shown in FIG. 8 . Elements of the electronic device 10 are not limited to those shown in FIG. 8 .

The communication unit 810 may communicate with an external device. The communication unit 810 may include at least one of a Wi-Fi chip, a Bluetooth chip, an NFC chip, and a wireless communication chip. The processor 850 may communicate with various external devices using the communication unit 810 .

The camera 820 may capture an image of the product. The captured image may be saved. Also, the camera 820 may capture at least one image of the product based on a level representing a maintenance progress state of the product.

The output unit 830 may display status information representing a maintenance state of elements identified in the product image and guide information for guiding maintenance of the selected element. The output unit 830 may include a display panel or a speaker.

The memory 840 is an electronic device 10 for acquiring an image of a product, displaying status information indicating maintenance status of elements identified in the image, and displaying guide information guiding maintenance for the selected element. It may store software or programs for executing the operation method of

The memory 840 is a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg SD or XD memory, etc.), RAM (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (ROM, Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic It may include at least one type of storage medium among a disk and an optical disk.

The processor 850 may execute programs stored in the memory 840, read data or files stored in the memory 840, or store new data or files in the memory 840. The processor 850 may display state information indicating a maintenance state of elements identified in the image and guide information for guiding maintenance of the selected element by executing instructions stored in the memory 840 .

The processor 850 may obtain an image of the product. For example, a product may refer to an object requiring work such as assembly and maintenance.

For example, the processor 850 may obtain an image of a product through a camera 820 in an electric car device. For another example, the processor 850 may obtain an image of the product through the external camera 820 connected to the electronic device 10 . Here, the external camera 820 may capture maintenance progress of the product in real time and transmit the captured image to the electronic device 10 .

For example, the processor 850 may acquire at least one image of the product based on a level indicating a maintenance progress state of the product. For example, a level indicating a maintenance progress state may be classified according to a maintenance completion rate.

Based on a result of identifying a plurality of elements in the image, the processor 850 may display status information indicating a maintenance state of the plurality of elements through the output unit 830 .

For example, the processor 850 may detect a plurality of objects in an image. The electronic device 10 may identify a plurality of elements corresponding to a plurality of objects based on shape information or label information of each of the plurality of objects. The processor 850 may obtain state information indicating a maintenance state of a plurality of elements based on at least one of an assembly relationship between a plurality of elements, external appearance information of the plurality of elements, and use history information of the plurality of elements. .

For example, the processor 850 may determine the priority of at least one element requiring maintenance among a plurality of elements based on state information of the plurality of elements. The processor 850 may display state information of at least one element through the output unit 830 based on the priority of the at least one element.

For example, the processor 850 may display state information including at least one of a name of each of a plurality of elements, appearance information, and use history information through the output unit 830 .

The processor 850 may receive an input for selecting a first element from among a plurality of elements through the output unit 830 based on state information.

For example, the output unit 830 may classify and display elements that require maintenance, elements that have completed maintenance, and elements that do not require maintenance on an image. For example, the output unit 830 may display elements according to a display method corresponding to elements that require maintenance, elements that have completed maintenance, and elements that do not require maintenance. For example, the display method may include a method of differently displaying colors for each type. For example, the processor 850 may receive an input for selecting a first element among elements requiring maintenance on the image. For another example, the processor 850 may receive an input for selecting an icon representing state information of the first element.

The output unit 830 may display first guide information for guiding maintenance of the first element.

For example, the output unit 830 may display first guide information including at least one of a name of the first element, a replacement period, and a maintenance method.

For example, when the maintenance of the first element is completed, the output unit 830 determines the maintenance of the second element requiring subsequent maintenance after the maintenance of the first element, based on the state information of the plurality of elements. Second guide information for guiding may be displayed.

For example, the output unit 830 may display state information of a plurality of elements and first guide information of a first element on one screen on an image.

For example, the output unit 830 may display information about a maintenance progress status and expected maintenance time according to the maintenance progress of the first element.

For example, the output unit 830 may display a first execution icon for accessing schedule information of a mechanic who maintains a product. Also, the output unit 830 may display a second execution icon for accessing contact information of a customer who has requested maintenance of the product.

Those skilled in the art related to the present embodiment may know that other general-purpose components may be further included in addition to the components shown in FIG. 8 .

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, the scope of the present invention should not be limited to the described embodiments and should not be defined, but should be defined by not only the claims to be described later, but also those equivalent to these claims.

Claims

acquiring an image of the product;

based on a result of identifying a plurality of elements in the image, displaying status information indicating a maintenance state of the plurality of elements;

receiving an input for selecting a first element from among the plurality of elements based on the state information; and

and displaying first guide information for guiding maintenance of the first element.
According to claim 1,

Acquiring the image is

obtaining the image of the product through a camera in the electronic device or an external camera connected to the electronic device; and

and acquiring at least one image of the product based on a level representing a maintenance progress state of the product.
According to claim 1,

The step of displaying status information indicating the maintenance status of the plurality of elements,

detecting a plurality of objects in the image;

identifying the plurality of elements corresponding to the plurality of objects based on shape information or label information of each of the plurality of objects; and

Acquiring the state information indicating a maintenance state of the plurality of elements based on at least one of an assembly relationship between the plurality of elements, external appearance information of the plurality of elements, and use history information of the plurality of elements. Including, a method of operating an electronic device.
According to claim 1,

The step of displaying status information indicating the maintenance status of the plurality of elements,

determining a priority of at least one element requiring maintenance from among the plurality of elements, based on the status information of the plurality of elements; and

and displaying state information of the at least one element based on the priority of the at least one element.
According to claim 1,

The step of displaying status information indicating the maintenance status of the plurality of elements,

and displaying the state information including at least one of a name of each of the plurality of elements, appearance information, and use history information.
According to claim 1,

The step of displaying first guide information for guiding the maintenance of the first element,

and displaying the first guide information including at least one of a name of the first element, a replacement cycle, and a maintenance method.
According to claim 1,

When the maintenance of the first element is completed, based on the state information of the plurality of elements, second guide information for guiding maintenance of the second element requiring subsequent maintenance work after the maintenance work of the first element is provided. A method of operating an electronic device, further comprising displaying.
According to claim 1,

The step of displaying first guide information for guiding the maintenance of the first element,

and displaying the state information of the plurality of elements and the first guide information of the first element on a single screen on the image.
According to claim 1,

The step of displaying first guide information for guiding the maintenance of the first element,

and displaying information about a maintenance progress status and expected maintenance time according to the maintenance progress of the first element.
According to claim 1,

displaying a first execution icon for accessing schedule information of a mechanic who maintains the product; and

The method of operating the electronic device further comprising at least one step of displaying a second execution icon for accessing contact information of a customer who has requested maintenance of the product.