If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
The Laboratory Automation Plug & Play (LAPP) framework is an over-arching reference architecture concept for the integration of robots in life science laboratories. The plug & play nature lies in the fact that manual configuration is not required, including the teaching of the robots. In this paper a digital twin (DT) based concept is proposed that outlines the types of information that must be provided for each relevant component of the system. In particular, for the devices interfacing with the robot, the robot positions must be defined beforehand in a device-attached coordinate system (CS) by the vendor. This CS must be detectable by the vision system of the robot by means of optical markers placed on the front side of the device. With that, the robot is capable of tending the machine by performing the pick-and-place type transportation of standard sample carriers. This basic use case is the primary scope of the LAPP-DT framework. The hardware scope is limited to simple benchtop and mobile manipulators with parallel grippers at this stage. This paper first provides an overview of relevant literature and state-of-the-art solutions, after which it outlines the framework on the conceptual level, followed by the specification of the relevant DT parameters for the robot, for the devices and for the facility. Finally, appropriate technologies and strategies are identified for the implementation.
This article is the second in the series Towards Robotic Laboratory Automation Plug & Play. The first paper outlined the concept in a high-level fashion and stated the fundamental goal for the Laboratory Automation Plug & Play (LAPP) framework: to provide a comprehensive, over-arching reference architecture and abstraction layer for robot-focused laboratory automation [
]. The top level in this context is the process representation and scheduling layer, which serves the orchestration of the laboratory workflows. This includes communication and control aspects considering various types of (semi-)automated laboratory devices (such as liquid handlers, plate storage, centrifuges, incubators and readers) as well as laboratory robots.
The framework aims to simplify the integration and set-up process for these types of robots by freeing the integrator/user from the burden of manually setting up the communication and teaching the robot motions. As a crucial part of the LAPP framework, the digital twin (DT) layer enables the storing and sharing of the robot-relevant information in regard to the various components of the system. Most importantly, the positions for arm control must be stored in a standardized fashion (encoded in the DT) and provided with the device out-of-the-box. Using these pieces of information, a mobile manipulator (MoMa) will be capable - after localizing the device with a fiducial marker - to tend the device in a plug & play manner.
The LAPP framework focuses at first on a specific type of MoMa that is now used by multiple solution providers for sample transportation in automated laboratories [
]. For this, the typical lab automation MoMa anatomy is considered. Such a robot consists of the following components:
A wheeled autonomous mobile base capable of simultaneous localization and mapping (SLAM)
A robot arm of four to seven degrees of freedom (DoF)
A vision system that aids the fine position detection for gripping and manipulation planning
A MoMa can be considered as the most complex robot system in our scope, which represents the broadest variety of components. As such, simpler robots (e.g., stationary or rail-mounted arms) can be derived from this set by omitting certain components (e.g., mobile navigation). On the use case side, the concept is developed at the first stage for simple pick-and-place type sample transportation and machine tending without any physical interaction with the equipment. Complex robot-equipment interactions will be elaborated in follow-up articles.
The paper starts with the Methods section, where the strategy for elaborating the concept is outlined. Then a section on Literature and the state-of-the-art provides an overview of the relevant solutions in regard to the vertical of robot-centered laboratory automation systems. In the section on Manual teaching of laboratory robots, the lowest level is addressed by reviewing how prevalent manual teaching approaches work. To conclude the literature and state-of-the-art review, the digital twin (DT) approach is reviewed, which will serve as the basis of the LAPP-DT proposal. In Teaching-free operation with the LAPP digital twin the concept is proposed, followed by identifying and discussing Challenges and key enablers. After that, key Implementation considerations are discussed and the pieces of the technological stack are outlined. Finally, Future work and next steps are introduced, and a Summary is provided.
The LAPP concept aims to provide an overview of the system components and their interfaces, outlining a comprehensive integration framework as a reference architecture model. One crucial aspect of this is the information representation layers (or digital twins), which can facilitate teaching-free robot integration. Both the over-arching LAPP framework and the DT layer (subject of this paper) are created and elaborated in a scalable and extendable manner throughout an evolutionary development process. This development approach can be outlined with the following bullet points:
The concept is first formulated in an abstract and high-level fashion, as published in [
]. It will then be concretized and adapted with the involvement of the professional community throughout continuous evolution.
The definition of the framework fundamentally focuses on the already existing and established building-block technologies but will be suitable to incorporate new developments, such as advances in collaborative robotics.
Initially, a limited set of typical laboratory tasks that are already subject to a certain level of automation/robotization will be considered. Later, as the point above suggests, new technologies will enable covering more complex tasks that the framework also must be able to incorporate.
Literature and the state-of-the-art
Manual teaching of laboratory robots
At the current state of technology, robotic manipulators in laboratory automation are mostly intended for transporting standard ANSI/SLAS-conform
Meets the Standards ANSI/SLAS 1-2004 through ANSI/SLAS 4-2004
microplates between different laboratory devices. To make this possible, in most cases, online teaching takes place when setting up the robotic system. This means that the positions are manually set either by moving the robot by hand or by jogging it with the controller. Fig. 1 shows the coordinate systems (frames) and positions that are typically used in laboratory robotics scenarios.
In the case of stationary robots, either the robot configuration is stored for each position in the form of joint values, or the position is directly prescribed in relation to the robot’s base CS (r in Fig. 1a). Between these taught positions, different types of movements are possible. Usually a sequence of intermediary approach positions are defined for the robot, before it reaches a final hand-over site with its gripper tool center point (TCP). This is to make sure that no collisions occur between the robot and its surroundings, including the device it is supposed to load with samples. A typical sequence of steps for a pick-and-place task from site 1 (s1) to site 2 (s2) is presented below, using the coordinate systems in Fig. 1a:
The robot starts in its home position (h)
Performs a MoveJ-type movement to ”untangle” itself and arrives at the standby configuration at the position (u)
Moves linearly (MoveL) to a device-approach position (s1d)
] (Fig. 1b m). This is necessary because the precision of the mobile base (Fig. 1b r) is not sufficient for the positions to be defined in a word-fixed CS (Fig. 1b w).
The robot itself is keeping track of its coordinate systems (frames) by maintaining a transfer tree, where relations between the frames are expressed. For example, the pose of the camera frame and the TCP are both known in relation to the robot’s base frame. When the arm is moving, the system regularly updates the corresponding chain of frames based on the kinematic model and the joint angles (forward kinematics).
The teaching process for marker-guided MoMas typically consists of the following steps, as deduced from the feature definitions for Astech Projects’ MoMa [
The operator drives the robot to the station and makes sure the marker is in the camera’s field of view
This pose of the mobile robot is stored as a point of interest (PoI) with the station’s name
The vision system determines the transformation from the camera frame to the marker frame:
The operator moves the arm to the hand-over position so that the TCP and the site coordinate systems align:
Based on the robot kinematics model, the relation between the camera frame and the TCP () is determined
The relation between the marker and the hand-over site is calculated:
Similar to stationary robots, the intermediary positions (ss and sd in Fig. 1b) can be taught additionally. The concept proposed in this paper provides a solution for eliminating the need for manual teaching and setup. For that, the standardized representation of the teaching positions and other relevant parameters based on the digital twin approach is presented in Section Teachingfree operation with the LAPP digital twin.
The digital twin approach
One of the most important aspects of the LAPP framework is that it provides an information layer of the assets of the laboratory. As proposed in [
], a LAPP-enabled laboratory device has a DT representation containing its parameters. These include both parameters that are constant and assigned when the device is manufactured as well as parameters that change during the operation of the device and that describe the current state. The approach of a virtual representation of physical entities aligns with the digital twin (DT) concept. In this Section, an overview of the DT concept is provided and the relevant nomenclature is specified. Originally, this concept was proposed by Michael Grieves at NASA [
] for product life cycle management purposes. Maintaining a virtual representation of a physical entity and implementing bi-directional data connections enables a variety of virtual operations ranging from modeling through testing to optimization. All these operations correspond to a specific sub-space of the digital twin. Generally, data are fed from the real space to the virtual space, and information is fed back alongside with processes. A DT accompanies the product throughout its entire life cycle and, according to the specific stage, can take up different forms. In parallel to the design phase of the product, a prototype of the DT is also developed, which is then instantiated for each physical product individually. The whole set of these instances make up the aggregate, and they all reside in the corresponding environment. The state of both the physical and the virtual entities are stored in the form of parameters. These can either be static (so-called prototype) parameters, which have the same value for each piece of the same product, or they can correspond to a specific physical specimen, in which case they are called instance parameters.
Since the proposal of the original concept, the DT notion has been adapted to a broad variety of use-cases. Jones et al. [
] provide a systematic review of the related literature by means of a thematic analysis and characterization. The paper also consolidates the terminology in regard to the terminology that will be followed in the context of the present article. Jones et al. also specify the parameters that are often used for digital twins, which are summarized in Table 1.
]. To adapt the off-line-created robot program to the physical workcell, calibration and manual adjustments are needed in on-line mode most of the time. To bridge this gap, they define the so-called Digital twin closeness as the measure of the ”geometric difference between the digital and physical counterparts of digital twins for robotic workcells”. Their DT comprises the models in Table 2. The LAPP-DT concept presented in this paper follows an approach that shares many aspects with Tipary’s DT. See Section Teaching-free operation with the LAPP digital twin.
], the LAPP framework aims to make the whole set-up process of laboratory robots fully automatic - all the way from map generation with simultaneous localization and mapping (SLAM) to determining the above-mentioned coordinate frame transformations. According to the LAPP approach, the positions (s, ss, sd in Fig. 1b) must be defined in relation to the marker frame (m in Fig. 1b) in cartesian space. To make this possible, first the workpiece CS must be defined. Since the most common workpiece in this context is an ANSI/SLAS-conform microplate [
As discussed above, in the LAPP framework the positions are defined in relation to the marker frame in cartesian space and not in the robot’s CS in joint space. This enables a robot-independent implementation of the movements thanks to the fact that the positions belong to a certain site of a workstation or device and not to one specific robot. This makes sharing the positions between multiple robots possible. Ultimately, it can be assumed that a certain model of a laboratory device always has the same geometry, i.e., the hand-over site is at a pre-defined location of the device. Supposing that the marker is also at a fixed and known position, the transformation T5 (as displayed in Fig. 1b can be deduced. The LAPP framework thus requires the device vendor to include the marker in the design of the device. The marker must be in a visible location, on the same side from where the robot must access it. To ensure compatibility, a specific marker type and size must be specified. The currently available laboratory MoMas [
ARTag, AprilTag and CALTag fiducial marker systems: comparison in a presence of partial marker occlusion and rotation.
ICINCO 2017 - Proceedings of the 14th International Conference on Informatics in Control, Automation and Robotics. vol. 2. 9789897582646 SciTePress,
]). Although, only in an ideal world would all vendors commit to make their devices LAPP-compliant. Especially for the launching and ramp-up period, the database will rely more on crowd-sourcing. For this, the DT prototype taxonomy must be provided as an open repository, with clearly defined contribution and admission mechanisms. Also, individual deviations in the device geometry may occur, thus the possibility for adjusting the DT instance by the means of calibration must also be kept open. See Section Challenges and key enablers.
Table 3Parameters of the device’s digital twin prototype. CS markings defined in 1
]. After that, in ideal circumstances (i.e., with calibrated geometries), the pre-defined robotic actions can be performed right away, without requiring any manual input from the integrator or operator. This is the functionality referred to as plug & play. As a part of the sequence presented in [
], the coordinate transformations are determined and the digital twin instances are parameterized, as presented in Table 4.
Table 4The plug & play setup sequence, markings of Fig. 1b used
DT parameter type
During the autonomous room discovery procedure, the map is generated
Simultaneously, the approximate device positions are detected with the markers. Since d and m are already connected by the DT prototype of the device, and the robot is at the PoI d is now defined in w.
The hand-over site position is taken from the DT prototype of the device
If necessary, this position can be overridden (calibrated) and stored in the DT instance of the device.
The robot kinematics can be re-calibrated and stored in the DT instance of the robot
The relation between the frames and positions in such a system depend on each other in that one frame is defined as a transformation of another frame. This means that the frames can be represented in a graph structure, such as in Fig. 4, where the nodes are the frames and the edges are the transformations. The fact that fairly long chains of interlinked frames can form means that the uncertainties of each transformation can accumulate. These uncertainties are inevitably present both in the form of mechanical imperfections causing deviations from ideal geometries and in the inherent inaccuracies of the metrology systems [
Probabilistic robotics (intelligent robotics and autonomous agents series).
vol. 45. 02622016232005
]. These must be taken into account by storing the uncertainty for each transformation in the digital twin. Also, a requirement in regard to precision must be specified for the specific robot action. This enables the process controller to match the requirement with the actual precision and determine if the specific system can perform the requested action.
It is also important to mention, that the parent-child relations in a transform tree can change, if a more accurate definition is available. For example, the following case can be considered: When the robot moves to a certain PoI, the system can assume the approximate positions of the devices as defined in relation to the map , as shown in Fig. 4a. However, the vision system can deliver a far more accurate relation between the robot and the marker (), which, along with can be used for motion planning of the pick & place subtask. For this, a sub-tree from the robot frame (r) can be temporarily created, as presented in Fig. 4b.
Challenges and key enablers
Traditional laboratory robots are usually taught in joint-space, which means that the robot configuration is stored directly in the form of joint values for each position. This results in the fact that the accuracy of the robot’s kinematic model is not of crucial importance. On the contrary, when the positions are specified in world coordinates, the robot controller must perform the inverse kinematics calculations to determine the corresponding joint values. For this, the robot’s geometry must be precisely modeled, e.g., by Denavit-Hartenberg parameters or by the Unified Robot Description Format (URDF). These are specified initially for each robot model during the design process. Following the DT notation, these data correspond to the DT prototype of the robot model. Due to manufacturing imperfections, however, each piece of a finished robot has slightly different geometries in reality, which must be readjusted by the means of calibration. These new parameters are stored in the DT instance for the certain robot. To achieve the sub-millimeter precision required for efficient and reliable plate manipulation, the possibility must be kept open for the robot to be re-calibrated.
Another distinction is between online vs. offline teaching. Online teaching means that the positions are taught directly on the physical robot by moving its end-effector to the desired positions manually. In this case, the robot’s repeatability is more important than its absolute accuracy; what matters is that the robot is consistent even if systematic error is present. Offline teaching means that the positions are defined without the physical robot, e.g., in the case of the LAPP approach by the device vendor. In this case, the absolute accuracy is just as important as the repeatability, since consistency is not sufficient if the position is off. Therefore, the precision requirements towards a LAPP-enabled robot must be formulated adequately. For digital twin-based offline teaching scenarios in particular, the digital twin closeness to measure such precision is defined by Tipary et al. [
Besides kinematics, the mechanics of the robot also play an important role in the robot’s precision. One such effect is that no mechanical system is perfectly rigid but flexible and compliant to a certain degree. Reasons behind this are on one hand the mechanical properties of materials: depending on a component’s shape and its elastic modulus, it bends under load. On the other hand, the joints of robots consist of bearings and gear systems, which might have mechanical play or backlash. These two effects result in the whole structure diverging from the ideal position when under the effect of internal and external forces, which could be calculated by pure kinematics. These effects must be either mechanically minimized or countered by a suitable controller.
Calibration might also be necessary with regard to the devices’ geometries and the positions of the hand-over sites. As described in Section Teaching-free operation with the LAPP digital twin, this transformation () can be fetched from the DT prototype, but overridden (calibrated) on-demand in the DT instance.
To make the plug & play functionality possible, a series of complementary technologies must be consolidated and integrated into a comprehensive stack. To this end, different competing and overlapping solutions were evaluated. Since compatibility is a crucial factor, established and widespread standardized solutions are preferred.
], setting up a robot-based laboratory automation system begins with representing the process on the high-level to enable execution on an all-round process controller. For this purpose, representation formats, such as the Business Process Model and Notation (BPMN) and the Laboratory Open Protocol (LabOP) will be evaluated (See Future work. Below this level, the robot-specific actions must be represented in the appropriate abstraction level, for which the LAPP RAPs are proposed and are subject to future work. On the low-level - i.e., on the robot’s side - a versatile robot control framework is needed that is capable of incorporating advanced techniques for perception, navigation and motion planning. The Robot Operating System (ROS) proves to fulfill these requirements.
To enable the communication between the different components and levels of the system, a suitable interoperability protocol is needed. The communication protocol developed by the Association Consortium Standardization in Lab Automation (SiLA) provides such a solution specifically for laboratory automation purposes. The organization currently endeavors to extend the robot-related functionality, including the implementation of a SiLA-ROS bridge [
] is identified as a suitable solution. Ontologies for labware, laboratory devices and laboratory robots must be identified, adapted and extended to fulfill the information sharing functionality that is outlined in this paper.
The LAPP framework, including the LAPP-DT, is at this point a conceptual proposal that is elaborated in the course of an article series entitled Towards Laboratory Automation Plug & Play. This present paper is the second in this series, which takes a step in concretizing the proposal with the focus on robotics. The series follows a top-down approach, which means that the follow-up articles will progress towards the lower technical levels of the framework and provide implementation examples, feasibility studies and comparable benchmarking.
As such, the robotic activity representations (RARs) are identified as the next work package to be elaborated. RARs are considered as one piece of information that must be represented in the digital twin. In general, the digital twin properties must be defined for each type of component in detail, including the identification of suitable standardized representation formats. Further work must focus on the technical implementation by utilizing the technologies discussed in the Implementation considerations section. Also, the challenges covered in the section on Challenges and key enablers must be addressed.
The first stage of the implementation is limited to benchtop manipulators and ground-bound mobile manipulators from the hardware side, both with simple parallel grippers. On the use-case side, the scope is focused on pick-and-place type transportation of standardized objects, such as ANSI/SLAS-microplates. Next stages of the development may broaden this scope in regard to both aspects. On one hand, a higher diversity in lab automation robotic hardware can be covered by including emerging technologies. As such, miniature MoMas that drive on the bench or on tracks at ceiling level [
] or soft grippers, open up new possibilities to manipulate objects that are fragile and/or have complex geometries. Ultimately, flexible marker-less manipulation can be achieved by the means of advanced perception, such as 3D object detection [
]. These techniques make it possible to detect the pose of the workpiece and to plan the manipulation movement without pre-taught positions. Besides that, the planning also must consider the constraint space containing, inter alia, the obstacles around the working area. Similarly to the other DT parameters, an appropriate standardized format must be identified to store and share this information between the different components of the system.
As a part of the Laboratory Automation Plug & Play framework, a digital-twin-based approach was proposed that ensures teaching-free setup for laboratory robots. An overview of the state-of-the-art approach showed that current MoMa solutions mostly rely on manually teaching the positions with relation to a fiducial marker. A review of the relevant literature on the digital twin approach was provided to consolidate the nomenclature. The LAPP-DT concept was elaborated on the conceptual level by defining a relative position representation framework as a part of the digital twin of laboratory equipment. After identifying the challenges and the key enablers, specific technological building blocks were formulated, considering the implementation of the framework. A roadmap was presented for the future steps in concretizing the conceptual framework and providing proof-of-concept implementations.
Declaration of competing interest
Ádám Wolf and Stefan Romeder-Finger are employees of Baxalta Innovations GmbH, a Takeda company, Vienna, Austria. Stefan Romeder-Finger is a stockholder in Takeda Pharmaceutical Company Limited.
The authors declare the following financial interests/personal relationships which may be considered as potential competing interests:
Adam Wolf reports financial support was provided by Baxalta Innovations GmbH. Adam Wolf reports financial support was provided by Obuda University Doctoral School of Applied Informatics and Applied Mathematics. Peter Galambos reports financial support was provided by National Research, Development and In- novation Fund of Hungary. Karoly Szell reports financial support was provided by ational Research, Development and In- novation Fund of Hungary. Adam Wolf reports a relationship with Takeda Manufacturing Austria AG that includes: employment. Stefan Romeder-Finger reports a relationship with Baxalta Innovations GmbH that includes: employment. Stefan Romeder-Finger reports a relationship with Takeda Pharmaceutical Company Limited that includes: equity or stocks.
This work was funded by Baxalta Innovations GmbH, a Takeda company.
This work was supported by the Doctoral School of Applied Informatics and Applied Mathematics, Óbuda University.
Péter Galambos and Károly Széll thankfully acknowledge the financial support of this work by the project no. 2019-1.3.1-KK-2019-00007 implemented with the support provided from the National Research, Development and Innovation Fund of Hungary, financed under the 2019-1.3.1-KK funding scheme. Péter Galambos is a Bolyai Fellow of the Hungarian Academy of Sciences.
Towards robotic laboratory automation plug & play: the ”LAPP” framework.
ARTag, AprilTag and CALTag fiducial marker systems: comparison in a presence of partial marker occlusion and rotation.
ICINCO 2017 - Proceedings of the 14th International Conference on Informatics in Control, Automation and Robotics. vol. 2. 9789897582646 SciTePress,