CN112883754A

CN112883754A - Auxiliary operation system of forklift

Info

Publication number: CN112883754A
Application number: CN201911195695.2A
Authority: CN
Inventors: 张东珉
Original assignee: BMW Brilliance Automotive Ltd
Current assignee: BMW Brilliance Automotive Ltd
Priority date: 2019-11-29
Filing date: 2019-11-29
Publication date: 2021-06-01
Anticipated expiration: 2039-11-29
Also published as: CN112883754B

Abstract

The present disclosure relates to a secondary operating system for a forklift. Fork truck includes fork truck arm and control cabin, and this auxiliary operating system includes: a camera mounted at a center of a forklift arm tip of a forklift, configured to capture an image ahead of the forklift arm tip; an electronic device coupled with the camera, configured to: receiving an image from a camera; and displaying the image from the camera in real time. This fork truck's auxiliary operation system can help the operator to pinpoint the fork truck arm, avoids damaging the fork truck arm or by the fork object.

Description

Auxiliary operation system of forklift

Technical Field

The present disclosure relates to fork lift trucks, and more particularly, to fork lift trucks having an auxiliary operating system, auxiliary operating systems for fork lift trucks, and related methods of operation.

Background

When operating a forklift, it is difficult for an operator to accurately operate the forklift arm into a small insertion opening, such as a small gap, so that the forklift arm or an object to be forked is easily damaged due to inaccuracy in operation.

Disclosure of Invention

According to some embodiments of the present disclosure, there is provided an auxiliary operating system of a forklift including a forklift arm and an operating cabin, the auxiliary operating system including: a camera mounted at a center of a forklift arm tip of a forklift, configured to capture an image ahead of the forklift arm tip; an electronic device coupled with the camera, configured to: receiving an image from a camera; and displaying the image from the camera in real time.

According to some embodiments of the disclosure, the electronic device is further configured to: receiving, via a user interface on an electronic device, a first user input to enter a locate mode; in response to receiving a first user input to enter a locate mode: displaying a reference center point on a user interface, the reference center point being a geometric center point of a display area for display of an image of one camera; displaying a lock user interface element on the user interface, the lock user interface element having a lock region center point that is displayed.

According to some embodiments of the disclosure, the electronic device is further configured to: receiving, via a user interface on an electronic device, a first user operation input by a user operating the locked user interface element; wherein the first user operational input comprises one or more of: moving; amplifying; and shrink.

According to some embodiments of the disclosure, the electronic device is further configured to: receiving a second user operation input of a user operating the locking user interface element through a user interface on the electronic equipment, wherein the second user operation input is used for locking the image content covered by the locking user interface element; in response to receiving a second user operation input, when the image content of the displayed camera changes, the locking user interface element is caused to move along with the image content locked by the locking user interface element.

According to some embodiments of the disclosure, the electronic device is further configured to: receiving a third user input to complete the positioning via a user interface on the electronic device; in response to receiving a third user input, changing a state of the locking user interface element to indicate that the positioning is complete.

According to some embodiments of the disclosure, the electronic device is further configured to: receiving, via a user interface on the electronic device, a fourth user input to enter an assistance mode; in response to receiving the fourth user input, entering an auxiliary mode, wherein in the auxiliary mode, if the locked region center point of the locked user interface element is mismatched with the reference center point, the state of the locked user interface element is changed to indicate the mismatch.

According to some embodiments of the present disclosure, the electronic device is further configured to, prior to receiving a third user input to complete the positioning via a user interface on the electronic device: and when the center point of the locking area of the locking user interface element is matched with the reference center point, outputting a prompt of successful positioning through a user interface.

According to some embodiments of the present disclosure, the lock user interface element is a rectangular box, and the lock area center point is a geometric center point of the rectangular box.

According to some embodiments of the disclosure, the electronic device is further configured to: in a positioning mode, changing the state of the locking user interface element to indicate that positioning is complete by changing the rectangle represented by the dashed line to the rectangle represented by the solid line; and in the secondary mode, changing the state of the locked user interface element to indicate the mismatch by changing the rectangular box represented by the solid line to the rectangular box represented by the dashed line.

According to some embodiments of the present disclosure, the reference center point is represented by a dashed circle and the lock region center point of the lock user interface element is represented by a solid circle.

According to some embodiments of the disclosure, the electronic device is further configured to: receiving a user selection of a connection via a user interface on the electronic device; and connecting the electronic equipment and the camera in the selected connection mode.

According to some embodiments of the present disclosure, the forklift has two forklift arms, the electronic device is further configured to: images shot by the cameras on the two forklift arms are displayed in the two display areas respectively.

According to some embodiments of the present disclosure, the forklift has two forklift arms, the electronic device is further configured to: and respectively setting corresponding locking user interface elements for images shot by the two cameras respectively displayed through the user interface.

According to some embodiments of the present disclosure, the camera is mounted in a cavity provided inside a tip of the forklift arm for accommodating the camera and photographs via an opening of the cavity on an end surface of the tip.

Drawings

Fig. 1A is a schematic diagram of a forklift with an auxiliary operating system according to an embodiment of the present disclosure.

Fig. 1B shows a schematic view of a camera mounted forklift arm according to an embodiment of the disclosure.

Fig. 2A is a schematic view of a forklift arm for mounting a camera according to an embodiment of the disclosure.

Fig. 2B is a schematic view of a camera mounted forklift arm according to an embodiment of the disclosure.

Fig. 3 illustrates an exemplary flow chart of a method of forklift assisted operation performed by an electronic device according to an embodiment of the disclosure.

Fig. 4A-4H illustrate an exemplary graphical user interface relating to a method of forklift assisted operation performed by an electronic device, in accordance with an embodiment of the present disclosure.

FIG. 5 is a schematic diagram illustrating a general hardware environment in which an electronic device according to embodiments of the disclosure may be implemented.

Detailed Description

The following description is presented to enable any person skilled in the art to make and use the embodiments, and is provided in the context of a particular application and its requirements. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the described embodiments. Thus, the described embodiments are not limited to the embodiments shown, but are to be accorded the widest scope consistent with the principles and features disclosed herein.

In the present disclosure, the user interface may be an example of one or more graphical User Interfaces (UIs) used to implement various features, processes, or workflows. These GUIs can be presented on a variety of electronic devices including, but not limited to, laptop computers, desktop computers, computer terminals, television systems, tablet computers, e-book readers, and smart phones. One or more of these electronic devices may incorporate a touch-sensitive surface. The touch sensitive surface is capable of processing multiple simultaneous input points, including processing data regarding the pressure, degree, or location of each input point. This process can facilitate gestures using multiple fingers, including furling and swiping.

When reference is made in this disclosure to "selecting" a user interface element in a GUI, these terms are understood to include clicking or "hovering" over the user interface element using a mouse or other input device, or touching, tapping, or gesturing over the user interface element using one or more fingers or a stylus. The user interface elements may be virtual buttons, menus, selectors, switches, sliders, dismissors, knobs, thumbnails, links, icons, radial buttons, check boxes, and any other mechanism for receiving input from or providing feedback to a user.

SUMMARY

The utility model relates to an auxiliary operating system for fork truck, this auxiliary operating system is including installing the camera and the electronic equipment that is coupled with the camera in the terminal central authorities department of fork truck arm of fork truck, and the image in the terminal the place ahead of fork truck arm is shot to the camera, and electronic equipment acquires and shows the image that the camera was shot, and the operator of help fork truck observes the location of fork truck arm to operate to the fork truck arm according to this and will fork truck arm pinpoint to the position of aliging with the inserted hole.

The electronic device has a user interface via which an operator may interact with the electronic device to better operate the forklift arm with the auxiliary operating system provided by the present disclosure.

Fig. 1A is a schematic diagram of a forklift 100 with an auxiliary operating system according to an embodiment of the present disclosure.

As shown in fig. 1A, the forklift 100 includes an operator cabin 102 and a forklift arm 104. The forklift 100 also includes an auxiliary operating system including a camera 106 mounted at the end of the forklift arm 104 and electronics 108 mounted, for example, within the operator's compartment 102 or generally used by an operator within the operator's compartment. The camera 106 may capture images of the front of the end of the forklift arm and the electronic device 108 may display the images captured by the camera 106 in real time.

The operator can better operate the forklift arm to work according to the image/picture in front of the tail end of the forklift arm 104 displayed in real time by the electronic equipment, and the risk of damaging the forked object is reduced.

The camera 106 may be any camera device that can be used to take pictures and is suitably mounted at the end of the forklift arm 104. The camera 106 may be equipped with, for example, a WiFi or bluetooth module for wireless connection with the electronic device.

The forklift arm on the forklift can be one or more, and each forklift arm can be provided with a camera. In this embodiment, for convenience of description, a forklift has two forklift arms side by side on the left and right, and one camera is mounted on each forklift arm. FIG. 1B shows a camera 106₁And 106₂Schematic diagrams installed on the left and right forklift arms of a forklift.

In some embodiments, the electronic device 108 may be a forklift-integrated electronic device. In other embodiments, the electronic device 108 may be an electronic device mounted to a suitable location within the operating compartment of the forklift. The electronic device 108 may be a stand-alone electronic device or an electronic device, typically used by an operator within an operator compartment. The electronic device 108 may have a display and one or more processors. The electronic device 108 may be any electronic device that has an associated secondary operating application installed.

The electronic device 108 and the camera 106 may be connected by wire or wirelessly. One or more of the electronic device 108 and the camera head 106 may include an associated signal processing unit (not shown) or be associated with a separate signal processing unit. The signal processing unit processes, for example, an output signal of the camera 106 into a display signal displayed by the electronic device 108. The signal processing unit may also be an integrated processing unit in the forklift. The signal processing unit may be a device having data processing capability, such as a processor.

Fig. 2A is a schematic view of a forklift arm for mounting a camera according to an embodiment of the disclosure. Fig. 2A illustrates one of the forklift arms 104. As shown in fig. 2A, a cavity 202 is provided inside the end of the forklift arm 104 for receiving/mounting a camera. The cavity 202 has an opening 201 in the centre of the end face of the end of the forklift arm 104. The opening 201 is provided in the center of the end face so that the center point of the picture taken by the camera is on the center line of travel of the forklift arm to the front. Here, it is assumed that the cross section of the forklift arm is a uniform rectangle, but in reality, the cross section of the forklift arm may not be a uniform rectangle, and those skilled in the art can understand that the imaging center of the mounted camera may be as close as possible to the center line of travel of the forklift arm directly in front. This enables a good operation effect to be achieved by a simple structural design.

In fig. 2A, the opening 201 is shown as being rectangular in shape. It will be appreciated by those skilled in the art that the shape of the opening 201 may also be circular or any other suitable shape.

Fig. 2B is a schematic diagram of another example of mounting a camera on a forklift arm according to an embodiment of the disclosure.

As shown in fig. 2B, the camera 106 may also be mounted on the end face of the end of the forklift arm. For example, a camera 106 is mounted on the forklift arm end at the center of the end face of the forklift arm end. The camera head 106 may have a protective member (not shown) having an edge that does not extend beyond the edge of the end face of the forklift arm end to avoid damage to the camera head when the forklift arm 106 is in operation.

It will be appreciated by those skilled in the art that the camera head 106 may be mounted/secured within the cavity 202 or on the end face of the end of the forklift arm 104 in a variety of ways known in the art, such as with an adhesive layer, screw fastening members, snap-fit structures, and the like. And will not be described in detail herein.

Those skilled in the art will appreciate that whatever the manner in which the camera is mounted and the particular location of the mounting, the center of the camera's view will be as close as possible to the center line of travel of the forklift arm directly forward.

Fig. 3 illustrates an exemplary flow chart of a method 300 of forklift assisted operation performed by an electronic device according to an embodiment of the disclosure. Fig. 4A-4H illustrate associated graphical user interfaces.

As shown, the method 300 includes step 301: in the display mode, images from the camera are received and displayed in real time.

FIG. 4A shows a graphical user interface in display mode. When the user enters the display mode, for example, by clicking "display mode", the picture taken by the camera is displayed in real time on the user interface of the electronic apparatus. Images in front of the forklift arms shot by the cameras on the left forklift arm and the right forklift arm are displayed in the picture.

In this mode, the user can determine whether the forklift arm is aligned with the insertion opening by means of the displayed image. Since the camera is disposed at the center of the end of the forklift arm, if the insertion opening is in the center in the photographed image, it can be determined that the forklift arm is aligned with the insertion opening.

In actual operation, the operator generally moves the forklift arm to the vicinity of the insertion opening, but the operator may not be able to accurately align the insertion opening. For example, in a captured image, the insertion port is not located at the center of the screen but at any position above, below, to the left, or to the right of the screen. At this point, the operator can perform a more accurate positioning of the forklift arm by clicking on the "positioning mode".

As shown in fig. 3, the method 300 further includes step 303: receiving, via a user interface on the electronic device, a first user input to enter a positioning mode, and in response to receiving the first user input to enter the positioning mode: the reference center point and the locked user interface element are displayed on the user interface. Wherein the reference center point is a geometric center point of a display area for display of an image of one camera. The lock user interface element has a lock region center point displayed.

FIG. 4B illustrates an exemplary graphical user interface after entering a locate mode. The user may enter the "locate mode" by entering a first user input via the user interface, for example, by clicking the "locate mode". After entering the positioning mode, an operable lock user interface element indicated by a rectangular white dashed box and a lock region center point of the lock user interface element indicated by a white solid circle are displayed in each of the left and right image display regions for the two cameras. The lock region center point may be a geometric center point of the lock user interface element. At the same time, a reference center point of the entire image display area is also displayed, which is indicated by a white dotted circle.

The method 300 further comprises the step 305: a first user operational input of a user operating the lock user interface element is received via a user interface on the electronic device. The first user operation input may move, zoom in, and/or zoom out.

FIG. 4C illustrates the exemplary graphical user interface after receiving a first user operational input. The user may drag the locked user interface element, i.e., the rectangular white dashed box, to move it, and/or zoom the rectangular white dashed box, so that the rectangular white dashed box just overlaps the insertion opening, sized to just cover the insertion opening. In the operation process, the central point of the locking area of the locking user interface element is always positioned in the center of the locking user interface element along with the movement and the scaling of the rectangular white dotted line frame.

As shown, the solid white line circle (the locking region center point) and the dashed white line circle (the reference center point) do not coincide (or align/match), i.e., the forklift arm is not precisely aligned with the insertion opening.

The method 300 further comprises step 307: and receiving a second user operation input of the user for operating the locking user interface element through a user interface on the electronic equipment, wherein the second user operation input is used for locking the image content covered by the locking user interface element, and after the second user operation input is received, when the image content of the displayed camera is changed, the locking user interface element is enabled to move along with the locked image content.

In some embodiments, the user may lock the image content covered by the locking user interface element, such as an insertion port image, for example, by double-clicking or otherwise manipulating the locking user interface element. This means that when the operator then operates the forklift arm to move (e.g., to the right), causing the image of the insertion opening to move accordingly in the screen (e.g., also to the right), the locking user interface element (e.g., the rectangular white dashed box) that locks the image of the insertion opening will move accordingly, and the locking area center point (e.g., the white solid circle) in the locking user interface element will move simultaneously. This intuitively informs the operator whether his operation of the forklift arm is further aligning the insertion port.

It will be understood by those skilled in the art that the confirmation operation performed by the user on the lock user interface element is not limited to double-clicking, and for example, the confirmation operation may be performed by selecting a prompt for the prompt that causes the user interface to display whether the lock is being performed. In other embodiments, the confirmation operation may be performed by a long press for more than, for example, 3 seconds.

The method 300 further comprises step 309: receiving a third user input via a user interface on the electronic device to complete the positioning, and in response to receiving the third user input, changing a state of the locking user interface element to indicate that the positioning is complete.

When the operator moves the forklift arm, the picture shot by the camera is updated in real time, the position of the insertion opening in the picture displayed on the electronic equipment is also moved, and accordingly, the locking user interface element (for example, a rectangular white dotted frame) moves along with the movement of the image of the insertion opening. When the lock user interface element is moved to a position such that the lock region center point (e.g., a white solid line circle) coincides with the reference center point (e.g., a white dashed line circle). It will be appreciated that at this point the forklift arm is correctly aligned with the insertion opening. Fig. 4D shows the graphical user interface at this time. As shown, the white solid line circle coincides with the white dashed line circle.

In some embodiments, at this point, a prompt may also be displayed on the user interface to prompt the operator that the forklift arm has been properly aligned with the insertion opening.

The operator may now enter a third user input to complete the positioning via the user interface on the electronic device, for example by clicking the "positioning mode" again. In response to receiving the third user input, the electronic device may change a state of the lock user interface element to indicate that the positioning is complete. Fig. 4E shows an exemplary graphical user interface at this time. As shown in fig. 4E, the rectangular frame indicated by the broken line is changed to the rectangular frame indicated by the solid line. Those skilled in the art will appreciate that there are many implementations for changing the state of the locking user interface element, as long as the change in state is reflected, thereby alerting the operator of the positioning completion. For example, the rectangular frame indicated by the broken line may be changed to the rectangular frame indicated by the broken line. The positioning completion may also be indicated by a change in color of the rectangular box.

The method 300 further comprises step 311: receiving, via a user interface on the electronic device, a fourth user input to enter an assistance mode; in response to receiving the fourth user input, entering an auxiliary mode, wherein in the auxiliary mode, if the locked region center point of the locked user interface element is mismatched with the reference center point, the state of the locked user interface element is changed to indicate the mismatch, e.g., requiring repositioning.

After the positioning is completed, the operator can operate the forklift arm insertion slot. At this time, the operator may enter the assist mode by, for example, clicking "assist mode". While the operator operates the forklift arm to insert into the insertion opening, the screen shot by the camera is still updated in real time, the lock user interface element (for example, a rectangular white solid line frame) moves following the movement of the image of the insertion opening, and when the lock user interface element moves to a position such that the lock area center point (for example, a white solid line circle) deviates from the reference center point (for example, a white dotted line circle) due to an undesired operation, as shown in fig. 4F, for example, the deviation is indicated by changing the rectangular white solid line frame to the rectangular white solid line frame dotted line frame. In some embodiments, a prompt may also be displayed on the user interface to prompt the operator that the forklift arm has been misaligned from the insertion opening.

In some embodiments, the secondary operations application may also provide functionality for the user to select which wireless connection to connect the electronic device and the camera. For example, as shown in fig. 4G and 4H, a "connect" button and an icon representing a WiFi connection (fig. 4G) or a bluetooth connection (fig. 4H) are provided on the user interface. The operator may select the corresponding wireless connection mode by clicking "connect" in a graphical user interface representing the corresponding connection mode.

FIG. 5 is a schematic diagram illustrating a general hardware environment in which an electronic device 700 according to embodiments of the present disclosure may be implemented.

Referring now to fig. 5, a schematic diagram of an example of an electronic device 700 is shown. Electronic device 700 is only one example of a suitable electronic device and is not intended to suggest any limitation as to the scope of use or functionality of the embodiments described herein. In any event, electronic device 700 is capable of implementing and/or performing any of the functions set forth above.

In the electronic device 700, there is a computer system/server 7012 that is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known computing systems, environments, and/or configurations that may be suitable for use with computer system/server 7012 include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computer systems, distributed cloud computing environments that include any of the above systems or devices, and the like.

Computer system/server 7012 may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer system/server 7012 may be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.

As shown in fig. 5, the computer system/server 7012 in the electronic device 700 is shown in the form of a general purpose computing device. Components of computer system/server 7012 may include, but are not limited to: one or more processors or processing units 7016, a system memory 7028, and a bus 7018 that couples various system components including the system memory 7028 to the processing unit 7016.

The bus 7018 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, micro channel architecture (MAC) bus, enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, Peripheral Component Interconnect (PCI) bus, peripheral component interconnect express (PCIe), and Advanced Microcontroller Bus Architecture (AMBA).

Computer system/server 7012 typically includes a variety of computer system readable media. Such media can be any available media that is accessible by computer system/server 7012 and includes both volatile and nonvolatile media, removable and non-removable media.

The system memory 7028 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)30 and/or cache memory 7032. The computer system/server 7012 may also include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, a storage system 7034 may be provided for reading from and writing to non-removable, nonvolatile magnetic media (not shown, and commonly referred to as "hard drives"). Although not shown, a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from and writing to a removable, non-volatile optical disk (such as a CD-ROM, DVD-ROM, or other optical media) may be provided. In such cases, each can be connected to the bus 7018 by one or more data media interfaces. As will be further depicted and described below, memory 7028 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the present disclosure.

By way of example, and not limitation, programs/utilities 7040, including a set of (at least one) program modules 7042, as well as an operating system, one or more application programs, other program modules, and program data, may be stored in memory 7028. Each of the operating system, one or more application programs, other program modules, and program data or some combination thereof, may include an implementation of a network environment. Program modules 7042 generally perform the functions and/or methods in embodiments as described herein.

The computer system/server 7012 may also communicate with one or more external devices 7014 (such as a keyboard, pointing device, electronic device 7024, etc.), one or more devices that enable a user to interact with the computer system/server 7012, and/or any device (e.g., network card, modem, etc.) that enables the computer system/server 7012 to communicate with one or more other computing devices. Such communication may occur via an input/output (I/O) interface 22. Also, the computer system/server 7012 may communicate with one or more networks, such as a Local Area Network (LAN), a general Wide Area Network (WAN), and/or a public network (e.g., the internet), via the network adapter 20. As depicted, the network adapter 20 communicates with the other components of the computer system/server 7012 via a bus 7018. It should be appreciated that although not shown, other hardware and/or software components may be used in conjunction with the computer system/server 7012. Examples include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data archive storage systems, and the like.

The present disclosure may be embodied as systems, methods, and/or computer program products. The computer program product may include computer-readable storage medium(s) having computer-readable program instructions thereon for causing a processor to perform aspects of the present disclosure.

The computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device such as a punch card or an in-groove projection structure having instructions stored thereon, and any suitable combination of the foregoing. A computer-readable storage medium as used herein is not to be interpreted as a transitory signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device via a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in the respective computing/processing device.

Computer-readable program instructions for carrying out operations of the present disclosure may be assembler instructions, Instruction Set Architecture (ISA) instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, the electronic circuitry, including, for example, programmable logic circuitry, Field Programmable Gate Arrays (FPGAs), or Programmable Logic Arrays (PLAs), may execute the computer-readable program instructions in order to perform aspects of the present disclosure by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry.

Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable medium having stored thereon the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a process for electronic device execution, such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

It should also be understood by those skilled in the art that various operations illustrated as sequential in the embodiments of the present disclosure do not necessarily have to be performed in the illustrated order. The order of operations may be adjusted as desired by those skilled in the art. One skilled in the art may also add more operations or omit some of them as desired.

The description of the various embodiments of the present disclosure has been presented for purposes of illustration but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles of the embodiments, the practical application, or technical improvements to the techniques found in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An auxiliary operating system for a forklift, the forklift including a forklift arm and an operating cabin, the auxiliary operating system comprising:

a camera mounted at a center of a forklift arm tip of a forklift, configured to capture an image ahead of the forklift arm tip;

an electronic device coupled with the camera, configured to:

receiving an image from a camera;

and displaying the image from the camera in real time.

2. The secondary operating system of claim 1, wherein the electronic device is further configured to:

receiving, via a user interface on an electronic device, a first user input to enter a locate mode;

in response to receiving a first user input to enter a locate mode:

displaying a reference center point on a user interface, the reference center point being a geometric center point of a display area for display of an image of one camera;

displaying a lock user interface element on the user interface, the lock user interface element having a lock region center point that is displayed.

3. The secondary operating system of claim 2, wherein the electronic device is further configured to:

receiving, via a user interface on an electronic device, a first user operation input by a user operating the locked user interface element;

wherein the first user operational input comprises one or more of:

moving;

amplifying; and

and (5) shrinking.

4. The secondary operating system of claim 3, wherein the electronic device is further configured to:

receiving a second user operation input of a user operating the locking user interface element through a user interface on the electronic equipment, wherein the second user operation input is used for locking the image content covered by the locking user interface element;

in response to receiving a second user operation input, when the image content of the displayed camera changes, the locking user interface element is caused to move along with the image content locked by the locking user interface element.

5. The secondary operating system of claim 4, wherein the electronic device is further configured to:

receiving a third user input to complete the positioning via a user interface on the electronic device;

in response to receiving a third user input, changing a state of the locking user interface element to indicate that the positioning is complete.

6. The secondary operating system of claim 5, wherein the electronic device is further configured to:

receiving, via a user interface on the electronic device, a fourth user input to enter an assistance mode;

in response to receiving the fourth user input, enter an assist mode,

wherein in the auxiliary mode, if the locked region center point of the locked user interface element is mismatched with the reference center point, the state of the locked user interface element is changed to indicate the mismatch.

7. The secondary operating system of claim 5, wherein the electronic device is further configured to, prior to receiving a third user input to complete the positioning via the user interface on the electronic device:

and when the center point of the locking area of the locking user interface element is matched with the reference center point, outputting a prompt of successful positioning through a user interface.

8. The secondary operating system of claims 2-7, wherein the lock user interface element is a rectangular box with a lock region center point being a geometric center point of the rectangular box.

9. The secondary operating system of claim 6, wherein the electronic device is further configured to:

in a positioning mode, changing the state of the locking user interface element to indicate that positioning is complete by changing the rectangle represented by the dashed line to the rectangle represented by the solid line; and is

In the secondary mode, the state of the locked user interface element is changed to indicate the mismatch by changing the rectangular box represented by the solid line to the rectangular box represented by the dashed line.

10. The secondary operating system of any of claims 2-9, wherein the reference center point is represented by a dashed circle and the lock region center point of the lock user interface element is represented by a solid circle.

11. The secondary operating system of any one of claims 1-10, wherein the electronic device is further configured to:

receiving a user selection of a connection via a user interface on the electronic device; and

and connecting the electronic equipment and the camera in a selected connection mode.

12. The secondary operating system of any one of claims 1-11, wherein the forklift has two forklift arms, the electronic device further configured to:

images shot by the cameras on the two forklift arms are displayed in the two display areas respectively.

13. The secondary operating system of any one of claims 2-11, wherein the forklift has two forklift arms, the electronic device further configured to:

and respectively setting corresponding locking user interface elements for images shot by the two cameras respectively displayed through the user interface.

14. The secondary operating system according to any one of claims 1 to 11, wherein the camera is mounted in a cavity provided inside a tip end of the forklift arm for accommodating the camera and photographing is performed via an opening of the cavity on an end face of the tip end.