Advertisement
Rapid Communication| Volume 28, ISSUE 2, P82-88, April 2023

Download started.

Ok

Towards robotic laboratory automation Plug & play: Survey and concept proposal on teaching-free robot integration with the lapp digital twin

Open AccessPublished:January 13, 2023DOI:https://doi.org/10.1016/j.slast.2023.01.003

      Abstract

      The Laboratory Automation Plug & Play (LAPP) framework is an over-arching reference architecture concept for the integration of robots in life science laboratories. The plug & play nature lies in the fact that manual configuration is not required, including the teaching of the robots. In this paper a digital twin (DT) based concept is proposed that outlines the types of information that must be provided for each relevant component of the system. In particular, for the devices interfacing with the robot, the robot positions must be defined beforehand in a device-attached coordinate system (CS) by the vendor. This CS must be detectable by the vision system of the robot by means of optical markers placed on the front side of the device. With that, the robot is capable of tending the machine by performing the pick-and-place type transportation of standard sample carriers. This basic use case is the primary scope of the LAPP-DT framework. The hardware scope is limited to simple benchtop and mobile manipulators with parallel grippers at this stage. This paper first provides an overview of relevant literature and state-of-the-art solutions, after which it outlines the framework on the conceptual level, followed by the specification of the relevant DT parameters for the robot, for the devices and for the facility. Finally, appropriate technologies and strategies are identified for the implementation.

      Keywords

      Introduction

      This article is the second in the series Towards Robotic Laboratory Automation Plug & Play. The first paper outlined the concept in a high-level fashion and stated the fundamental goal for the Laboratory Automation Plug & Play (LAPP) framework: to provide a comprehensive, over-arching reference architecture and abstraction layer for robot-focused laboratory automation [
      • Wolf A.
      • Wolton D.
      • Trapl J.
      • Janda J.
      • Romeder-Finger S.
      • Gatternig T.
      • Farcet J.-B.
      • Galambos P.
      • Széll K.
      Towards robotic laboratory automation plug & play: the ”LAPP” framework.
      ]. The top level in this context is the process representation and scheduling layer, which serves the orchestration of the laboratory workflows. This includes communication and control aspects considering various types of (semi-)automated laboratory devices (such as liquid handlers, plate storage, centrifuges, incubators and readers) as well as laboratory robots.
      The framework aims to simplify the integration and set-up process for these types of robots by freeing the integrator/user from the burden of manually setting up the communication and teaching the robot motions. As a crucial part of the LAPP framework, the digital twin (DT) layer enables the storing and sharing of the robot-relevant information in regard to the various components of the system. Most importantly, the positions for arm control must be stored in a standardized fashion (encoded in the DT) and provided with the device out-of-the-box. Using these pieces of information, a mobile manipulator (MoMa) will be capable - after localizing the device with a fiducial marker - to tend the device in a plug & play manner.
      The LAPP framework focuses at first on a specific type of MoMa that is now used by multiple solution providers for sample transportation in automated laboratories [
      • Kleine-Wechelmann S.
      • Bastiaanse K.
      • Freundel M.
      • Becker-Asano C.
      Designing the mobile robot Kevin for a life science laboratory.
      ,

      Biosero acceleration lab. [cited 2021 Oct 18]., URL https://biosero.com/integrations/acceleration-lab/.

      , ]. For this, the typical lab automation MoMa anatomy is considered. Such a robot consists of the following components:
      • A wheeled autonomous mobile base capable of simultaneous localization and mapping (SLAM)
      • A robot arm of four to seven degrees of freedom (DoF)
      • A vision system that aids the fine position detection for gripping and manipulation planning
      A MoMa can be considered as the most complex robot system in our scope, which represents the broadest variety of components. As such, simpler robots (e.g., stationary or rail-mounted arms) can be derived from this set by omitting certain components (e.g., mobile navigation). On the use case side, the concept is developed at the first stage for simple pick-and-place type sample transportation and machine tending without any physical interaction with the equipment. Complex robot-equipment interactions will be elaborated in follow-up articles.
      The paper starts with the Methods section, where the strategy for elaborating the concept is outlined. Then a section on Literature and the state-of-the-art provides an overview of the relevant solutions in regard to the vertical of robot-centered laboratory automation systems. In the section on Manual teaching of laboratory robots, the lowest level is addressed by reviewing how prevalent manual teaching approaches work. To conclude the literature and state-of-the-art review, the digital twin (DT) approach is reviewed, which will serve as the basis of the LAPP-DT proposal. In Teaching-free operation with the LAPP digital twin the concept is proposed, followed by identifying and discussing Challenges and key enablers. After that, key Implementation considerations are discussed and the pieces of the technological stack are outlined. Finally, Future work and next steps are introduced, and a Summary is provided.

      Methods

      The LAPP concept aims to provide an overview of the system components and their interfaces, outlining a comprehensive integration framework as a reference architecture model. One crucial aspect of this is the information representation layers (or digital twins), which can facilitate teaching-free robot integration. Both the over-arching LAPP framework and the DT layer (subject of this paper) are created and elaborated in a scalable and extendable manner throughout an evolutionary development process. This development approach can be outlined with the following bullet points:
      • The concept is first formulated in an abstract and high-level fashion, as published in [
        • Wolf A.
        • Wolton D.
        • Trapl J.
        • Janda J.
        • Romeder-Finger S.
        • Gatternig T.
        • Farcet J.-B.
        • Galambos P.
        • Széll K.
        Towards robotic laboratory automation plug & play: the ”LAPP” framework.
        ]. It will then be concretized and adapted with the involvement of the professional community throughout continuous evolution.
      • The definition of the framework fundamentally focuses on the already existing and established building-block technologies but will be suitable to incorporate new developments, such as advances in collaborative robotics.
      • Initially, a limited set of typical laboratory tasks that are already subject to a certain level of automation/robotization will be considered. Later, as the point above suggests, new technologies will enable covering more complex tasks that the framework also must be able to incorporate.

      Literature and the state-of-the-art

      Manual teaching of laboratory robots

      At the current state of technology, robotic manipulators in laboratory automation are mostly intended for transporting standard ANSI/SLAS-conform
      Meets the Standards ANSI/SLAS 1-2004 through ANSI/SLAS 4-2004
      microplates between different laboratory devices. To make this possible, in most cases, online teaching takes place when setting up the robotic system. This means that the positions are manually set either by moving the robot by hand or by jogging it with the controller. Fig. 1 shows the coordinate systems (frames) and positions that are typically used in laboratory robotics scenarios.
      Fig. 1
      Fig. 1Coordinate systems in the LAPP environment, d: device CS, g: robot’s tool center point (TCP), h: home position for robot m: fiducial marker for device, r: robot’s base, PoI: base pose for station, s: hand-over site (plate nest) of device, sd: device approach position, ss: site approach position, T0...T6: relational frame definitions (T0: world-to-PoI, T1: camera-to-marker, T2: TCP-to-site, T3: camera-to-TCP, T4: marker-to-site, T5: device-to-marker, T6: world-to-device), u: stand-by position for robot, w: world. Lines: Orange: live robot-level, Green: stored in the LAPP-DT, Dashed: inaccurate transformation (originating from base odometry), Solid: accurate transformation (originating from robot kinematics, marker detection or DT-stored positions)
      In the case of stationary robots, either the robot configuration is stored for each position in the form of joint values, or the position is directly prescribed in relation to the robot’s base CS (r in Fig. 1a). Between these taught positions, different types of movements are possible. Usually a sequence of intermediary approach positions are defined for the robot, before it reaches a final hand-over site with its gripper tool center point (TCP). This is to make sure that no collisions occur between the robot and its surroundings, including the device it is supposed to load with samples. A typical sequence of steps for a pick-and-place task from site 1 (s1) to site 2 (s2) is presented below, using the coordinate systems in Fig. 1a:
      • The robot starts in its home position (h)
      • Performs a MoveJ-type movement to ”untangle” itself and arrives at the standby configuration at the position (u)
      • Moves linearly (MoveL) to a device-approach position (s1d)
      • MoveL to a site-approach position (s1s)
      • MoveL to the final site hand-over position (s1)
      • Grips the plate
      • MoveL back to s1s
      • MoveL back to s1d
      • MoveL to s2d (not displayed)
      • MoveL to s2s (not displayed)
      • MoveL to s2 (not displayed)
      • Releases the plate
      • MoveL back to s2s
      • MoveL back to s2d
      • MoveL to u and returns to standby/ready state
      As described in [
      • Wolf A.
      • Wolton D.
      • Trapl J.
      • Janda J.
      • Romeder-Finger S.
      • Gatternig T.
      • Farcet J.-B.
      • Galambos P.
      • Széll K.
      Towards robotic laboratory automation plug & play: the ”LAPP” framework.
      ], in the case of mobile manipulators, on the other hand, the positions are defined in relation to a fiducial marker, which is in most cases an optical augmented reality (AR) marker [
      • Garrido-Jurado S.
      • Muñoz Salinas R.
      • Madrid-Cuevas F.J.
      • Marín-Jiménez M.J.
      Automatic generation and detection of highly reliable fiducial markers under occlusion.
      ] (Fig. 1b m). This is necessary because the precision of the mobile base (Fig. 1b r) is not sufficient for the positions to be defined in a word-fixed CS (Fig. 1b w).
      The robot itself is keeping track of its coordinate systems (frames) by maintaining a transfer tree, where relations between the frames are expressed. For example, the pose of the camera frame and the TCP are both known in relation to the robot’s base frame. When the arm is moving, the system regularly updates the corresponding chain of frames based on the kinematic model and the joint angles (forward kinematics).
      The teaching process for marker-guided MoMas typically consists of the following steps, as deduced from the feature definitions for Astech Projects’ MoMa [

      feature_definitions/ch/unitelabs · master · SiLA2 / sila_base · GitLab. URL https://gitlab.com/SiLA2/sila_base/-/tree/master/feature_definitions/ch/unitelabs.

      ] and from the Fraunhofer IPA’s Kevin Webinar [] (Markings of Fig. 1b are used.):
      • The operator drives the robot to the station and makes sure the marker is in the camera’s field of view
      • This pose of the mobile robot is stored as a point of interest (PoI) with the station’s name T0(wPoI)
      • The vision system determines the transformation from the camera frame to the marker frame: T1(cm)
      • The operator moves the arm to the hand-over position so that the TCP and the site coordinate systems align: T2(g=s)
      • Based on the robot kinematics model, the relation between the camera frame and the TCP (T3(cg)) is determined
      • The relation between the marker and the hand-over site is calculated: T4(ms)
      Similar to stationary robots, the intermediary positions (ss and sd in Fig. 1b) can be taught additionally. The concept proposed in this paper provides a solution for eliminating the need for manual teaching and setup. For that, the standardized representation of the teaching positions and other relevant parameters based on the digital twin approach is presented in Section Teachingfree operation with the LAPP digital twin.

      The digital twin approach

      One of the most important aspects of the LAPP framework is that it provides an information layer of the assets of the laboratory. As proposed in [
      • Wolf A.
      • Wolton D.
      • Trapl J.
      • Janda J.
      • Romeder-Finger S.
      • Gatternig T.
      • Farcet J.-B.
      • Galambos P.
      • Széll K.
      Towards robotic laboratory automation plug & play: the ”LAPP” framework.
      ], a LAPP-enabled laboratory device has a DT representation containing its parameters. These include both parameters that are constant and assigned when the device is manufactured as well as parameters that change during the operation of the device and that describe the current state. The approach of a virtual representation of physical entities aligns with the digital twin (DT) concept. In this Section, an overview of the DT concept is provided and the relevant nomenclature is specified. Originally, this concept was proposed by Michael Grieves at NASA [
      • Grieves M.
      • Vickers J.
      Digital twin: mitigating unpredictable, undesirable emergent behavior in complex systems.
      ] for product life cycle management purposes. Maintaining a virtual representation of a physical entity and implementing bi-directional data connections enables a variety of virtual operations ranging from modeling through testing to optimization. All these operations correspond to a specific sub-space of the digital twin. Generally, data are fed from the real space to the virtual space, and information is fed back alongside with processes. A DT accompanies the product throughout its entire life cycle and, according to the specific stage, can take up different forms. In parallel to the design phase of the product, a prototype of the DT is also developed, which is then instantiated for each physical product individually. The whole set of these instances make up the aggregate, and they all reside in the corresponding environment. The state of both the physical and the virtual entities are stored in the form of parameters. These can either be static (so-called prototype) parameters, which have the same value for each piece of the same product, or they can correspond to a specific physical specimen, in which case they are called instance parameters.
      Since the proposal of the original concept, the DT notion has been adapted to a broad variety of use-cases. Jones et al. [
      • Jones D.
      • Snider C.
      • Nassehi A.
      • Yon J.
      • Hicks B.
      Characterising the digital twin: a systematic literature review.
      ] provide a systematic review of the related literature by means of a thematic analysis and characterization. The paper also consolidates the terminology in regard to the terminology that will be followed in the context of the present article. Jones et al. also specify the parameters that are often used for digital twins, which are summarized in Table 1.
      Table 1Parameters of the digital twin
      • Jones D.
      • Snider C.
      • Nassehi A.
      • Yon J.
      • Hicks B.
      Characterising the digital twin: a systematic literature review.
      .
      ParameterCharacteristics
      FormThe geometric structure of the entity Can contain dimensions, CS definitions and 3D models
      FunctionalityThe entity’s capabilities, movement or purpose, Definition of the control interface and parameters
      HealthState of the entity with respect to its ideal state
      LocationPosition and orientation (pose) With respect to the environment, layout or other entity
      ProcessActivities in which the entity is engaged, Scheduling parameters and models
      TimeDuration to complete an activity, Timestamp
      StateMeasured (live) values of entity and environment parameters
      PerformanceMeasured operation with respect to the optimal operation
      EnvironmentThe physical and virtual environment as a parameter
      MiscellaneousRequirements Qualification
      The DT concept is broadly applied for smart manufacturing use cases [
      • Rosen R.
      • Von Wichert G.
      • Lo G.
      • Bettenhausen K.D.
      About the importance of autonomy and digital twins for the future of manufacturing.
      ,
      • Negri E.
      • Fumagalli L.
      • Macchi M.
      A review of the roles of digital twin in CPS-based production systems.
      ,
      • Lu Y.
      • Liu C.
      • Wang K.I.K.
      • Huang H.
      • Xu X.
      Digital Twin-driven smart manufacturing: connotation, reference model, applications and research issues.
      ,
      • Schleich B.
      • Anwer N.
      • Mathieu L.
      • Wartzack S.
      Shaping the digital twin for design and production engineering.
      ], and there are also numerous examples of robotics applications, where the DT may have different goals. These range from the education of future engineers [
      • Verner I.
      • Cuperman D.
      • Fang A.
      • Reitman M.
      • Romm T.
      • Balikin G.
      Robot online learning through digital twin experiments: a weightlifting project.
      ,
      • Magrin C.E.
      • Del Conte G.
      • Todt E.
      Creating a digital twin as an open source learning tool for mobile robotics.
      ] through human-machine interactions [
      • Malik A.A.
      • Bilberg A.
      Digital twins of human robot collaboration in a production setting.
      ,
      • Pairet E.
      • Ardón P.
      • Liu X.
      • Lopes J.
      • Hastie H.
      • Lohan K.S.
      A digital twin for human-robot interaction.
      ,

      Baranyi P., Csapó A., Budai T., Wersényi G.. Introducing the concept of internet of digital reality-part I. Acta Polytechnica Hungarica 18(7):2021.

      ] and control [
      • Vu M.D.
      • Nguyen T.N.
      • My C.A.
      Design and implementation of a digital twin to control the industrial robot mitsubishi RV-12SD.
      ] all the way to programming [
      • Kuts V.
      • Sarkans M.
      • Otto T.
      • Tähemaa T.
      • Bondarenko Y.
      Digital twin: concept of hybrid programming for industrial robots - use case.
      ,
      • Burckhardt I.
      Laboratory automation in clinical microbiology.
      ]. Tipary and Erdös propose a design approach that utilizes parametric digital twins of robotic workcells for planning and programming [
      • Erdős G.
      • Paniti I.
      • Tipary B.
      Transformation of robotic workcells to digital twins.
      ,
      • Tipary B.
      • Erdős G.
      Generic development methodology for flexible robotic pick-and-place workcells based on Digital Twin.
      ]. To adapt the off-line-created robot program to the physical workcell, calibration and manual adjustments are needed in on-line mode most of the time. To bridge this gap, they define the so-called Digital twin closeness as the measure of the ”geometric difference between the digital and physical counterparts of digital twins for robotic workcells”. Their DT comprises the models in Table 2. The LAPP-DT concept presented in this paper follows an approach that shares many aspects with Tipary’s DT. See Section Teaching-free operation with the LAPP digital twin.
      Table 2Models in Tipary’s robot cell DT
      • Tipary B.
      ModelContent
      Kinematic modelGeometry, kinematic behavior and component relations
      Grasp modelWorkpiece - manipulator relation through the gripper
      Path modelManipulation sequence and manipulator motion
      Servo modelCondition-based manipulation (observation-based)
      Metrology modelProviding information to other models and planner tools (observation-based)
      Tolerance modelRepresent the tolerance stack-up of the operation to assess its feasibility

      Teaching-free operation with the LAPP digital twin

      As outlined in [
      • Wolf A.
      • Wolton D.
      • Trapl J.
      • Janda J.
      • Romeder-Finger S.
      • Gatternig T.
      • Farcet J.-B.
      • Galambos P.
      • Széll K.
      Towards robotic laboratory automation plug & play: the ”LAPP” framework.
      ], the LAPP framework aims to make the whole set-up process of laboratory robots fully automatic - all the way from map generation with simultaneous localization and mapping (SLAM) to determining the above-mentioned coordinate frame transformations. According to the LAPP approach, the positions (s, ss, sd in Fig. 1b) must be defined in relation to the marker frame (m in Fig. 1b) in cartesian space. To make this possible, first the workpiece CS must be defined. Since the most common workpiece in this context is an ANSI/SLAS-conform microplate [

      ANSI/SLAS. ANSI SLAS 4-2004 (R2012): microplate standards, well positions2004;.

      ], this is taken as a basis. In Fig. 2, the axes are derived, as overlaid on a schematically drawing, which is based on the ANSI/SLAS standard [

      ANSI/SLAS. ANSI SLAS 4-2004 (R2012): microplate standards, well positions2004;.

      ].
      Fig. 2
      Fig. 2Definition of the plate CS (Overlaid on
      [

      ANSI/SLAS. ANSI SLAS 4-2004 (R2012): microplate standards, well positions2004;.

      ]
      ) (US projection). Viewing from an upper view, the origin of the CS (o) is placed to the centerpoint of the plate, whereas the height is set to the middle of the bottom rim (b). When looking at the plate from above with well A1 being on the top left, the x axis points to the right, the y upwards, and z outwards. The TCP for the gripper and the CS of the hand-over site are defined in a way that their origins align with the plate CS and the axes either align or are rotated 90 degrees along the z axis (e.g., in portrait mode gripping, as seen in ). The site CS is defined similarly, derived from the plate CS.
      Fig. 3
      Fig. 3The plate- gripper- and site coordinate systems align in landscape mode gripping
      As discussed above, in the LAPP framework the positions are defined in relation to the marker frame in cartesian space and not in the robot’s CS in joint space. This enables a robot-independent implementation of the movements thanks to the fact that the positions belong to a certain site of a workstation or device and not to one specific robot. This makes sharing the positions between multiple robots possible. Ultimately, it can be assumed that a certain model of a laboratory device always has the same geometry, i.e., the hand-over site is at a pre-defined location of the device. Supposing that the marker is also at a fixed and known position, the transformation T5 (as displayed in Fig. 1b can be deduced. The LAPP framework thus requires the device vendor to include the marker in the design of the device. The marker must be in a visible location, on the same side from where the robot must access it. To ensure compatibility, a specific marker type and size must be specified. The currently available laboratory MoMas [
      • Kleine-Wechelmann S.
      • Bastiaanse K.
      • Freundel M.
      • Becker-Asano C.
      Designing the mobile robot Kevin for a life science laboratory.
      ,

      Biosero acceleration lab. [cited 2021 Oct 18]., URL https://biosero.com/integrations/acceleration-lab/.

      , ] use different types of ArUco [
      • Garrido-Jurado S.
      • Muñoz Salinas R.
      • Madrid-Cuevas F.J.
      • Marín-Jiménez M.J.
      Automatic generation and detection of highly reliable fiducial markers under occlusion.
      ], AprilTag [
      • Sagitov A.
      • Shabalina K.
      • Sabirova L.
      • Li H.
      • Magid E.
      ARTag, AprilTag and CALTag fiducial marker systems: comparison in a presence of partial marker occlusion and rotation.
      ] and PI-Tag [
      • Bergamasco F.
      • Albarelli A.
      • Torsello A.
      Pi-Tag: a fast image-space marker design based on projective invariants.
      ] markers of different sizes. Assessing these options and selecting a candidate is subject to future work.
      It is also the vendor’s responsibility to provide the information shown in Table 3 as part of the digital twin prototype of the device (See Section The digital twin approach, and Table 2 in [
      • Wolf A.
      • Wolton D.
      • Trapl J.
      • Janda J.
      • Romeder-Finger S.
      • Gatternig T.
      • Farcet J.-B.
      • Galambos P.
      • Széll K.
      Towards robotic laboratory automation plug & play: the ”LAPP” framework.
      ]). Although, only in an ideal world would all vendors commit to make their devices LAPP-compliant. Especially for the launching and ramp-up period, the database will rely more on crowd-sourcing. For this, the DT prototype taxonomy must be provided as an open repository, with clearly defined contribution and admission mechanisms. Also, individual deviations in the device geometry may occur, thus the possibility for adjusting the DT instance by the means of calibration must also be kept open. See Section Challenges and key enablers.
      Table 3Parameters of the device’s digital twin prototype. CS markings defined in 1
      Piece of informationAs digital twin parameter
      (See Table 1)
      Robotic tasks, subtasks and motion sequences: e.g., transport by pick & placeFunctionality
      Type of workpiece: ANSI/SLAS microplate, Falcon tube etc.Functionality
      Location of the marker (m) in relation to the device CS (d)Form
      Simplified collision geometry of the device, defined in dForm
      Positions. e.g., Hand-over site s, Site approach ss, Device approach sdForm
      Precision requirementsMiscellaneous
      In general, DT parameters are distinguished in the LAPP framework as follows:
      • Prototype parameters are unified for all specimens of the same make and model. See Table 3
      • Instance parameters override and extend the prototype parameters with instance-specific data
        • -
          Static parameters remain unchanged during the entire lifecycle of the instance. E.g., serial number
        • -
          Volatile parameters represent the current state of the instance
          • low asterisk
            Observable properties represent real-time or semi real-time data streams. E.g., sensor streams, continuous progress updates
          • low asterisk
            Unobservable properties are updated only upon specific request. E.g., calibrated positions
      With the help of this information, a LAPP-enabled MoMa with a calibrated camera system will be capable of performing the setup fully autonomously, as presented in Fig. 1 of [
      • Wolf A.
      • Wolton D.
      • Trapl J.
      • Janda J.
      • Romeder-Finger S.
      • Gatternig T.
      • Farcet J.-B.
      • Galambos P.
      • Széll K.
      Towards robotic laboratory automation plug & play: the ”LAPP” framework.
      ]. After that, in ideal circumstances (i.e., with calibrated geometries), the pre-defined robotic actions can be performed right away, without requiring any manual input from the integrator or operator. This is the functionality referred to as plug & play. As a part of the sequence presented in [
      • Wolf A.
      • Wolton D.
      • Trapl J.
      • Janda J.
      • Romeder-Finger S.
      • Gatternig T.
      • Farcet J.-B.
      • Galambos P.
      • Széll K.
      Towards robotic laboratory automation plug & play: the ”LAPP” framework.
      ], the coordinate transformations are determined and the digital twin instances are parameterized, as presented in Table 4.
      Table 4The plug & play setup sequence, markings of Fig. 1b used
      DescriptionTransform.DT parameter typeDT instance
      During the autonomous room discovery procedure, the map is generatedwFormRoom (instance)
      Simultaneously, the approximate device positions are detected with the markers. Since d and m are already connected by the DT prototype of the device, and the robot is at the PoI d is now defined in w.T6(wd)LocationDevice (instance)
      The hand-over site position is taken from the DT prototype of the deviceT4(ms)FormDevice (prototype)
      If necessary, this position can be overridden (calibrated) and stored in the DT instance of the device.T4,cal(ms)FormDevice (instance)
      The robot kinematics can be re-calibrated and stored in the DT instance of the robotrgFormRobot (instance)
      The relation between the frames and positions in such a system depend on each other in that one frame is defined as a transformation of another frame. This means that the frames can be represented in a graph structure, such as in Fig. 4, where the nodes are the frames and the edges are the transformations. The fact that fairly long chains of interlinked frames can form means that the uncertainties of each transformation can accumulate. These uncertainties are inevitably present both in the form of mechanical imperfections causing deviations from ideal geometries and in the inherent inaccuracies of the metrology systems [
      • Thrun S.
      • Burgard W.
      • Fox D.
      Probabilistic robotics (intelligent robotics and autonomous agents series).
      ]. These must be taken into account by storing the uncertainty for each transformation in the digital twin. Also, a requirement in regard to precision must be specified for the specific robot action. This enables the process controller to match the requirement with the actual precision and determine if the specific system can perform the requested action.
      Fig. 4
      Fig. 4Coordinate frame transformations (Markings of are used.)
      It is also important to mention, that the parent-child relations in a transform tree can change, if a more accurate definition is available. For example, the following case can be considered: When the robot moves to a certain PoI, the system can assume the approximate positions of the devices as defined in relation to the map T6(wd), as shown in Fig. 4a. However, the vision system can deliver a far more accurate relation between the robot and the marker (T1(cm)), which, along with T4(ms) can be used for motion planning of the pick & place subtask. For this, a sub-tree from the robot frame (r) can be temporarily created, as presented in Fig. 4b.

      Challenges and key enablers

      Traditional laboratory robots are usually taught in joint-space, which means that the robot configuration is stored directly in the form of joint values for each position. This results in the fact that the accuracy of the robot’s kinematic model is not of crucial importance. On the contrary, when the positions are specified in world coordinates, the robot controller must perform the inverse kinematics calculations to determine the corresponding joint values. For this, the robot’s geometry must be precisely modeled, e.g., by Denavit-Hartenberg parameters or by the Unified Robot Description Format (URDF). These are specified initially for each robot model during the design process. Following the DT notation, these data correspond to the DT prototype of the robot model. Due to manufacturing imperfections, however, each piece of a finished robot has slightly different geometries in reality, which must be readjusted by the means of calibration. These new parameters are stored in the DT instance for the certain robot. To achieve the sub-millimeter precision required for efficient and reliable plate manipulation, the possibility must be kept open for the robot to be re-calibrated.
      Another distinction is between online vs. offline teaching. Online teaching means that the positions are taught directly on the physical robot by moving its end-effector to the desired positions manually. In this case, the robot’s repeatability is more important than its absolute accuracy; what matters is that the robot is consistent even if systematic error is present. Offline teaching means that the positions are defined without the physical robot, e.g., in the case of the LAPP approach by the device vendor. In this case, the absolute accuracy is just as important as the repeatability, since consistency is not sufficient if the position is off. Therefore, the precision requirements towards a LAPP-enabled robot must be formulated adequately. For digital twin-based offline teaching scenarios in particular, the digital twin closeness to measure such precision is defined by Tipary et al. [
      • Tipary B.
      • Erdős G.
      Generic development methodology for flexible robotic pick-and-place workcells based on Digital Twin.
      ] (see Section The digital twin approach).
      Besides kinematics, the mechanics of the robot also play an important role in the robot’s precision. One such effect is that no mechanical system is perfectly rigid but flexible and compliant to a certain degree. Reasons behind this are on one hand the mechanical properties of materials: depending on a component’s shape and its elastic modulus, it bends under load. On the other hand, the joints of robots consist of bearings and gear systems, which might have mechanical play or backlash. These two effects result in the whole structure diverging from the ideal position when under the effect of internal and external forces, which could be calculated by pure kinematics. These effects must be either mechanically minimized or countered by a suitable controller.
      Calibration might also be necessary with regard to the devices’ geometries and the positions of the hand-over sites. As described in Section Teaching-free operation with the LAPP digital twin, this transformation (T4) can be fetched from the DT prototype, but overridden (calibrated) on-demand in the DT instance.

      Implementation considerations

      To make the plug & play functionality possible, a series of complementary technologies must be consolidated and integrated into a comprehensive stack. To this end, different competing and overlapping solutions were evaluated. Since compatibility is a crucial factor, established and widespread standardized solutions are preferred.
      As described in [
      • Wolf A.
      • Wolton D.
      • Trapl J.
      • Janda J.
      • Romeder-Finger S.
      • Gatternig T.
      • Farcet J.-B.
      • Galambos P.
      • Széll K.
      Towards robotic laboratory automation plug & play: the ”LAPP” framework.
      ], setting up a robot-based laboratory automation system begins with representing the process on the high-level to enable execution on an all-round process controller. For this purpose, representation formats, such as the Business Process Model and Notation (BPMN) and the Laboratory Open Protocol (LabOP) will be evaluated (See Future work. Below this level, the robot-specific actions must be represented in the appropriate abstraction level, for which the LAPP RAPs are proposed and are subject to future work. On the low-level - i.e., on the robot’s side - a versatile robot control framework is needed that is capable of incorporating advanced techniques for perception, navigation and motion planning. The Robot Operating System (ROS) proves to fulfill these requirements.
      To enable the communication between the different components and levels of the system, a suitable interoperability protocol is needed. The communication protocol developed by the Association Consortium Standardization in Lab Automation (SiLA) provides such a solution specifically for laboratory automation purposes. The organization currently endeavors to extend the robot-related functionality, including the implementation of a SiLA-ROS bridge [] and the unification of the feature definitions [

      feature_definitions/ch/unitelabs · master · SiLA2 / sila_base · GitLab. URL https://gitlab.com/SiLA2/sila_base/-/tree/master/feature_definitions/ch/unitelabs.

      ].
      Besides that, a suitable platform is needed for the digital twin representation that can incorporate the necessary parameters. The Asset Administration Shell [

      Details of the Asset Administration Shell. Part 1 - The exchange of information between partners in the value chain of Industrie 4.0 (Version 1.0). [cited 2021 June 09]. URL https://www.plattform-i40.de/PI40/Redaktion/DE/Downloads/Publikation/2018-verwaltungsschale-im-detail.html.

      ] is identified as a suitable solution. Ontologies for labware, laboratory devices and laboratory robots must be identified, adapted and extended to fulfill the information sharing functionality that is outlined in this paper.

      Future work

      The LAPP framework, including the LAPP-DT, is at this point a conceptual proposal that is elaborated in the course of an article series entitled Towards Laboratory Automation Plug & Play. This present paper is the second in this series, which takes a step in concretizing the proposal with the focus on robotics. The series follows a top-down approach, which means that the follow-up articles will progress towards the lower technical levels of the framework and provide implementation examples, feasibility studies and comparable benchmarking.
      As such, the robotic activity representations (RARs) are identified as the next work package to be elaborated. RARs are considered as one piece of information that must be represented in the digital twin. In general, the digital twin properties must be defined for each type of component in detail, including the identification of suitable standardized representation formats. Further work must focus on the technical implementation by utilizing the technologies discussed in the Implementation considerations section. Also, the challenges covered in the section on Challenges and key enablers must be addressed.
      The first stage of the implementation is limited to benchtop manipulators and ground-bound mobile manipulators from the hardware side, both with simple parallel grippers. On the use-case side, the scope is focused on pick-and-place type transportation of standardized objects, such as ANSI/SLAS-microplates. Next stages of the development may broaden this scope in regard to both aspects. On one hand, a higher diversity in lab automation robotic hardware can be covered by including emerging technologies. As such, miniature MoMas that drive on the bench or on tracks at ceiling level [] and drones [
      • Kim D.
      • Oh P.Y.
      Lab automation drones for mobile manipulation in high throughput systems.
      ] open up new dimensions regarding flexibility and they also require new approaches concerning navigation. On the other hand, advanced end effectors, such as five-finger [

      Pharmaceuticals & biomedical - shadow robot company. URL https://www.shadowrobot.com/pharmaceuticals-biomedical/.

      ] or soft grippers, open up new possibilities to manipulate objects that are fragile and/or have complex geometries. Ultimately, flexible marker-less manipulation can be achieved by the means of advanced perception, such as 3D object detection [
      • Vincze M.
      • Bajones M.
      • Suchi M.
      • Wolf D.
      • Weiss A.
      • Fischinger D.
      • Da La Puente P.
      Learning and detecting objects with a mobile robot to assist older adults in their homes.
      ] and tactile sensing [
      • Gomes D.F.
      • Lin Z.
      • Luo S.
      GelTip: a finger-shaped optical tactile sensor for robotic manipulation.
      ,
      • Thalhammer S.
      • Leitner M.
      • Patten T.
      • Vincze M.
      PyraPose: feature pyramids for fast and accurate object pose estimation under domain shift.
      ]. These techniques make it possible to detect the pose of the workpiece and to plan the manipulation movement without pre-taught positions. Besides that, the planning also must consider the constraint space containing, inter alia, the obstacles around the working area. Similarly to the other DT parameters, an appropriate standardized format must be identified to store and share this information between the different components of the system.

      Summary

      As a part of the Laboratory Automation Plug & Play framework, a digital-twin-based approach was proposed that ensures teaching-free setup for laboratory robots. An overview of the state-of-the-art approach showed that current MoMa solutions mostly rely on manually teaching the positions with relation to a fiducial marker. A review of the relevant literature on the digital twin approach was provided to consolidate the nomenclature. The LAPP-DT concept was elaborated on the conceptual level by defining a relative position representation framework as a part of the digital twin of laboratory equipment. After identifying the challenges and the key enablers, specific technological building blocks were formulated, considering the implementation of the framework. A roadmap was presented for the future steps in concretizing the conceptual framework and providing proof-of-concept implementations.

      Declaration of competing interest

      Ádám Wolf and Stefan Romeder-Finger are employees of Baxalta Innovations GmbH, a Takeda company, Vienna, Austria. Stefan Romeder-Finger is a stockholder in Takeda Pharmaceutical Company Limited.
      The authors declare the following financial interests/personal relationships which may be considered as potential competing interests:
      Adam Wolf reports financial support was provided by Baxalta Innovations GmbH. Adam Wolf reports financial support was provided by Obuda University Doctoral School of Applied Informatics and Applied Mathematics. Peter Galambos reports financial support was provided by National Research, Development and In- novation Fund of Hungary. Karoly Szell reports financial support was provided by ational Research, Development and In- novation Fund of Hungary. Adam Wolf reports a relationship with Takeda Manufacturing Austria AG that includes: employment. Stefan Romeder-Finger reports a relationship with Baxalta Innovations GmbH that includes: employment. Stefan Romeder-Finger reports a relationship with Takeda Pharmaceutical Company Limited that includes: equity or stocks.

      Acknowledgements

      This work was funded by Baxalta Innovations GmbH, a Takeda company.
      This work was supported by the Doctoral School of Applied Informatics and Applied Mathematics, Óbuda University.
      Péter Galambos and Károly Széll thankfully acknowledge the financial support of this work by the project no. 2019-1.3.1-KK-2019-00007 implemented with the support provided from the National Research, Development and Innovation Fund of Hungary, financed under the 2019-1.3.1-KK funding scheme. Péter Galambos is a Bolyai Fellow of the Hungarian Academy of Sciences.

      References

        • Wolf A.
        • Wolton D.
        • Trapl J.
        • Janda J.
        • Romeder-Finger S.
        • Gatternig T.
        • Farcet J.-B.
        • Galambos P.
        • Széll K.
        Towards robotic laboratory automation plug & play: the ”LAPP” framework.
        SLAS Technol. 2021; https://doi.org/10.1016/J.SLAST.2021.11.003
        • Kleine-Wechelmann S.
        • Bastiaanse K.
        • Freundel M.
        • Becker-Asano C.
        Designing the mobile robot Kevin for a life science laboratory.
        RO-MAN 2022 - 31st IEEE International Conference on Robot and Human Interactive Communication: Social, Asocial, and Antisocial Robots. 2022: 870-875https://doi.org/10.1109/RO-MAN53752.2022.9900786
      1. Biosero acceleration lab. [cited 2021 Oct 18]., URL https://biosero.com/integrations/acceleration-lab/.

      2. Robotic lab assistant - astech projects. URL https://astechprojects.co.uk/laboratory/robotic-lab-assistant/.

        • Garrido-Jurado S.
        • Muñoz Salinas R.
        • Madrid-Cuevas F.J.
        • Marín-Jiménez M.J.
        Automatic generation and detection of highly reliable fiducial markers under occlusion.
        Pattern Recognit. 2014; 47: 2280-2292https://doi.org/10.1016/j.patcog.2014.01.005
      3. feature_definitions/ch/unitelabs · master · SiLA2 / sila_base · GitLab. URL https://gitlab.com/SiLA2/sila_base/-/tree/master/feature_definitions/ch/unitelabs.

      4. (224) Webinar Kevin - YouTube. URL https://www.youtube.com/watch?v=HfZCNi841Oo&t=27s.

        • Grieves M.
        • Vickers J.
        Digital twin: mitigating unpredictable, undesirable emergent behavior in complex systems.
        Transdiscip Perspect Complex Syst. 2016; : 85-113https://doi.org/10.1007/978-3-319-38756-7_4
        • Jones D.
        • Snider C.
        • Nassehi A.
        • Yon J.
        • Hicks B.
        Characterising the digital twin: a systematic literature review.
        CIRP J Manuf Sci Technol. 2020; 29: 36-52https://doi.org/10.1016/j.cirpj.2020.02.002
        • Rosen R.
        • Von Wichert G.
        • Lo G.
        • Bettenhausen K.D.
        About the importance of autonomy and digital twins for the future of manufacturing.
        IFAC-PapersOnLine. 2015; 28: 567-572https://doi.org/10.1016/j.ifacol.2015.06.141
        • Negri E.
        • Fumagalli L.
        • Macchi M.
        A review of the roles of digital twin in CPS-based production systems.
        Procedia Manuf. 2017; 11: 939-948https://doi.org/10.1016/J.PROMFG.2017.07.198
        • Lu Y.
        • Liu C.
        • Wang K.I.K.
        • Huang H.
        • Xu X.
        Digital Twin-driven smart manufacturing: connotation, reference model, applications and research issues.
        Robot Comput-Integrat Manuf. 2020; 61https://doi.org/10.1016/J.RCIM.2019.101837
        • Schleich B.
        • Anwer N.
        • Mathieu L.
        • Wartzack S.
        Shaping the digital twin for design and production engineering.
        CIRP Annal - Manuf Technol. 2017; 66: 141-144https://doi.org/10.1016/J.CIRP.2017.04.040
        • Verner I.
        • Cuperman D.
        • Fang A.
        • Reitman M.
        • Romm T.
        • Balikin G.
        Robot online learning through digital twin experiments: a weightlifting project.
        Lect Note Netw Syst. 2018; 22: 307-314https://doi.org/10.1007/978-3-319-64352-6_29
        • Magrin C.E.
        • Del Conte G.
        • Todt E.
        Creating a digital twin as an open source learning tool for mobile robotics.
        2021 Latin American Robotics Symposium, 2021 Brazilian Symposium on Robotics, and 2021 Workshop on Robotics in Education, LARS-SBR-WRE 2021. 2021: 13-18https://doi.org/10.1109/LARS/SBR/WRE54079.2021.9605457
        • Malik A.A.
        • Bilberg A.
        Digital twins of human robot collaboration in a production setting.
        Procedia Manuf. 2018; 17: 278-285https://doi.org/10.1016/J.PROMFG.2018.10.047
        • Pairet E.
        • Ardón P.
        • Liu X.
        • Lopes J.
        • Hastie H.
        • Lohan K.S.
        A digital twin for human-robot interaction.
        ACM/IEEE International Conference on Human-Robot Interaction. 2019-March. 2019: 372https://doi.org/10.1109/HRI.2019.8673015
      5. Baranyi P., Csapó A., Budai T., Wersényi G.. Introducing the concept of internet of digital reality-part I. Acta Polytechnica Hungarica 18(7):2021.

        • Vu M.D.
        • Nguyen T.N.
        • My C.A.
        Design and implementation of a digital twin to control the industrial robot mitsubishi RV-12SD.
        Mech Mach Sci. 2021; 113 MMS: 422-432https://doi.org/10.1007/978-3-030-91892-7_40
        • Kuts V.
        • Sarkans M.
        • Otto T.
        • Tähemaa T.
        • Bondarenko Y.
        Digital twin: concept of hybrid programming for industrial robots - use case.
        ASME International Mechanical Engineering Congress and Exposition, Proceedings (IMECE). 2B-2019. 2020https://doi.org/10.1115/IMECE2019-10583
        • Burckhardt I.
        Laboratory automation in clinical microbiology.
        Bioengineering. 2018; 5: 102https://doi.org/10.3390/bioengineering5040102
        • Erdős G.
        • Paniti I.
        • Tipary B.
        Transformation of robotic workcells to digital twins.
        CIRP Annal. 2020; 69: 149-152https://doi.org/10.1016/J.CIRP.2020.03.003
        • Tipary B.
        • Erdős G.
        Generic development methodology for flexible robotic pick-and-place workcells based on Digital Twin.
        Robot Comput-Integrat Manuf. 2021; 71: 102140https://doi.org/10.1016/J.RCIM.2021.102140
        • Tipary B.
        Assembly plan calibration using sensor networks. 2022 (Ph.D. thesis)
      6. ANSI/SLAS. ANSI SLAS 4-2004 (R2012): microplate standards, well positions2004;.

        • Sagitov A.
        • Shabalina K.
        • Sabirova L.
        • Li H.
        • Magid E.
        ARTag, AprilTag and CALTag fiducial marker systems: comparison in a presence of partial marker occlusion and rotation.
        ICINCO 2017 - Proceedings of the 14th International Conference on Informatics in Control, Automation and Robotics. vol. 2. 9789897582646 SciTePress, 2017: 182-191https://doi.org/10.5220/0006478901820191
        • Bergamasco F.
        • Albarelli A.
        • Torsello A.
        Pi-Tag: a fast image-space marker design based on projective invariants.
        Mach Vis Appl. 2013; 24: 1295-1310https://doi.org/10.1007/s00138-012-0469-6
      7. SiLA2 / sila_ros · GitLab. URL https://gitlab.com/SiLA2/sila_ros.

      8. Details of the Asset Administration Shell. Part 1 - The exchange of information between partners in the value chain of Industrie 4.0 (Version 1.0). [cited 2021 June 09]. URL https://www.plattform-i40.de/PI40/Redaktion/DE/Downloads/Publikation/2018-verwaltungsschale-im-detail.html.

      9. ROVER® laboratory automation platform - FORMULATRIX®. URL https://formulatrix.com/liquid-handling-systems/rover-laboratory-automation/.

        • Kim D.
        • Oh P.Y.
        Lab automation drones for mobile manipulation in high throughput systems.
        2018 IEEE International Conference on Consumer Electronics, ICCE 2018. 2018-January. 2018: 1-5https://doi.org/10.1109/ICCE.2018.8326268
      10. Pharmaceuticals & biomedical - shadow robot company. URL https://www.shadowrobot.com/pharmaceuticals-biomedical/.

        • Vincze M.
        • Bajones M.
        • Suchi M.
        • Wolf D.
        • Weiss A.
        • Fischinger D.
        • Da La Puente P.
        Learning and detecting objects with a mobile robot to assist older adults in their homes.
        Lect Note Comput Sci (Includ Subser Lect Note Artif Intell Lect Note Bioinform). 2016; 9914 LNCS: 316-330https://doi.org/10.1007/978-3-319-48881-3_22/TABLES/2
        • Gomes D.F.
        • Lin Z.
        • Luo S.
        GelTip: a finger-shaped optical tactile sensor for robotic manipulation.
        IEEE International Conference on Intelligent Robots and Systems. 2020: 9903-9909https://doi.org/10.1109/IROS45743.2020.9340881
        • Thalhammer S.
        • Leitner M.
        • Patten T.
        • Vincze M.
        PyraPose: feature pyramids for fast and accurate object pose estimation under domain shift.
        2021 IEEE International Conference on Robotics and Automation (ICRA), Xi’an, China. 2021: 13909-13915https://doi.org/10.1109/ICRA48506.2021.9562108