Publications


2020

  1. Learning Grasping Points for Garment Manipulation in Robot-Assisted Dressing
    Fan Zhang, and Yiannis Demiris.

    IEEE International Conference on Robotics and Automation (ICRA), 2020

    Assistive robots have the potential to provide tremendous support for disabled and elderly people in their daily dressing activities. Recent studies on robot-assisted dressing usually simplify the setup of the initial robot configuration by manually attaching the garments on the robot end-effector and positioning them close to the user's arm. A fundamental challenge in automating such a process for robots is computing suitable grasping points on garments that facilitate robotic manipulation. In this paper, we address this problem by introducing a supervised deep neural network to locate a pre-defined grasping point on the garment, using depth images for their invariance to color and texture. To reduce the amount of real data required, which is costly to collect, we leverage the power of simulation to produce large amounts of labeled data. The network is jointly trained with synthetic datasets of depth images and a limited amount of real data. We introduce a robot-assisted dressing system that combines the grasping point prediction method, with a grasping and manipulation strategy which takes grasping orientation computation and robot-garment collision avoidance into account. The experimental results demonstrate that our method is capable of yielding accurate grasping point estimations. The proposed dressing system enables the Baxter robot to autonomously grasp a hospital gown hung on a rail, bring it close to the user and successfully dress the upper-body.

2019

  1. Probabilistic Real-Time User Posture Tracking for Personalized Robot-Assisted Dressing
    Fan Zhang, Antoine Cully, and Yiannis Demiris.

    IEEE Transactions on Robotics (T-RO)

    Robotic solutions to dressing assistance have the potential to provide tremendous support for elderly and disabled people. However, unexpected user movements may lead to dressing failures or even pose a risk to the user. Tracking such user movements with vision sensors is challenging due to severe visual occlusions created by the robot and clothes. We propose a probabilistic tracking method using Bayesian networks in latent spaces, which fuses robot end-effector positions and force information to enable camera-less and real-time estimation of the user postures during dressing. The latent spaces are created before dressing by modeling the user movements with a Gaussian Process Latent Variable Model, taking the user's movement limitations into account. We introduce a robot-assisted dressing system that combines our tracking method with hierarchical multi-task control to minimize the force between the user and the robot. The experimental results demonstrate the robustness and accuracy of our tracking method. The proposed method enables the Baxter robot to provide personalized dressing assistance in putting on a sleeveless jacket for users with (simulated) upper-body impairments.

2018

  1. Preoperative Optimization of the Surgical Robot Considering Internal Diversity of Workspace
    Zhiyuan Yan, Zhijiang Du, Fan Zhang, and Weidong Wang.

    Proceedings of the Institution of Mechanical Engineers, Part C: Journal of Mechanical Engineering Science

    Surgical robots have increased in popularity, and their performance is closely related to the robotic positioning before surgery. Many recent studies in preoperative planning have focused on the pose selection of the robot and the port placement. However, it is difficult to position the surgical robot simply based on experience. To solve this problem, the surgical workspace is subdivided into several subspaces with different weights. Global isotropy index and cooperation capability index are proposed to reflect the performance of the surgical robot and used as optimization functions. Particle swarm optimization is used to optimize the setup parameters. Based on different weight distributions, setup parameters can be automatically given and sent to the simulation system to display the setup and guide the robot positioning. The results show that the setup optimization considering the internal diversity of workspace is capable of satisfying the detailed requirements of robotic surgery and effectively guide the robotic surgery setup.

2017

  1. Personalized Robot-Assisted Dressing using User Modeling in Latent Spaces
    Fan Zhang, Antoine Cully, and Yiannis Demiris.

    2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

    Robots have the potential to provide tremendous support to disabled and elderly people in their everyday tasks, such as dressing. Many recent studies on robotic dressing assistance usually view dressing as a trajectory planning problem. However, the user movements during the dressing process are rarely taken into account, which often leads to the failures of the planned trajectory and may put the user at risk. The main difficulty of taking user movements into account is caused by severe occlusions created by the robot, the user, and the clothes during the dressing process, which prevent vision sensors from accurately detecting the postures of the user in real time. In this paper, we address this problem by introducing an approach that allows the robot to automatically adapt its motion according to the force applied on the robot's gripper caused by user movements. There are two main contributions introduced in this paper: 1) the use of a hierarchical multi-task control strategy to automatically adapt the robot motion and minimize the force applied between the user and the robot caused by user movements; 2) the online update of the dressing trajectory based on the user movement limitations modeled with the Gaussian Process Latent Variable Model in a latent space, and the density information extracted from such latent space. The combination of these two contributions leads to a personalized dressing assistance that can cope with unpredicted user movements during the dressing while constantly minimizing the force that the robot may apply on the user. The experimental results demonstrate that the proposed method allows the Baxter humanoid robot to provide personalized dressing assistance for human users with simulated upper-body impairments.
  2. Preoperative Planning for the Multi-Arm Surgical Robot using PSO-GP-based Performance Optimization
    Fan Zhang, Zhiyuan Yan, and Zhijiang Du.

    2017 IEEE International Conference on Robotics and Automation (ICRA)

    For the robotically-assisted minimally invasive surgery, preoperative planning is essential towards assisting surgeons to prepare the intervention and to decide the best access to the surgical site. Many recent studies in preoperative planning have focused on the pose selection of the robot and the port placement. However, as such techniques cannot evaluate the performance of the multi-arm cooperation, their applications are constrained in real practise with multi-arm surgical robots. In this paper, the surgical workspace is divided and the subspaces are assigned with different weights to reflect the internal differences within the surgical workspace. We propose three metrics to evaluate the performance of the multi-arm surgical robot: Global Isotropy Index (GII) to measure the dexterity of one single robot arm; Cooperation Capability Index (CCI) to reflect the performance of the multi-arm cooperation; Minimum Distance Index (MDI) to describe the collision avoidance of the robotic arms. We also propose a combination of Particle Swarm Optimization (PSO) and Gaussian Process (GP) to locate the port placement and robot positioning. The proposed integrated PSO-GP-based optimization strategy is implemented on a three-arm surgical robot. Two sets of experiments are carried out to validate our method. The results demonstrate that the performance optimization strategy based on PSO-GP is capable of guiding surgeons to plan an intervention with the multi-arm surgical robot.

2016

  1. Preoperative Setup Planning for Robotic Surgery based on a Simulation Platform and Gaussian Process
    Fan Zhang, Zhiyuan Yan, and Zhijiang Du,

    2016 IEEE International Conference on Mechatronics and Automation (ICMA)

    For the robotically-assisted minimally invasive surgery, preoperative planning is essential towards assisting surgeons to prepare the intervention and to decide the best access to the surgical site. Many recent studies in preoperative planning cannot evaluate the performance of the multi-arm cooperation, and thus their applications are constrained in real practise with multi-arm surgical robots. In this paper, we establish a simulation platform of a three-arm surgical robot. We propose to use two objective functions, Global Isotropy Index (GII) and Cooperation Capability Index (CCI), to reflect the dexterity of a robot arm and the performance of the multi-arm cooperation respectively. We also propose to use Gaussian Process Regression (GPR) to locate the optimal port placement and robot positioning. Simulation experiments are carried out to validate our method. The results demonstrate that our proposed performance optimization strategy based on GP is capable to guide surgeons to plan an intervention with the multi-arm surgical robot.

2015

  1. An Under-Actuated Manipulation Controller based on Workspace Analysis and Gaussian Processes
    Fan Zhang, Yanyu Su, Xiang Zhang, Wei Dong, and Zhijiang Du,

    2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

    The kinematic modelling has been applied to many controllers of under-actuated manipulators. Most of these studies assume that the control process is conducted within the workspace. However, as such a kinematic model cannot describe the situations when the stable grasping is violated in the real environment, these controllers may fail unexpectedly. In this paper, we propose a combination of kinematics based Workspace Analysis (WA) and Gaussian Process Classification (GPC) to model the success rates of control actions in the theoretical workspace. We also use the Gaussian Process Regression (GPR) to model the residual between the prediction of the WA and the ground truth data. We then apply this integrated model, Gaussian Processes enhanced Workspace Analysis (GP-WA), into an optimal controller. The optimal controller is implemented on a planar under-actuated gripper with two three-phalanx fingers. Two sets of simulation experiments are carried out to validate our method. The results demonstrate that the optimal manipulation controller based on GP-WA achieves high control accuracy for manipulating a wide range of objects.