Goals and Purpose

  • safeguard the beta cluster in case of disaster by backing up key data sources
    • global cluster filesystem
    • etcd configuration DB
    • provisioning state at time of provision
    • kubernetes dump
  • automated - hands-off operation
  • data stored off-site for reliability
  • NOT intended as a user-service for lost files, etc.
    • intended to rebuild entire system

 

Design Considerations and Decision history

  • gluster geo-replication to remote
    • requires gluster deploy remotely
    • does not specifically target backup
      • filesystem-synchronization is not the same as backup
    • Was not implemented - decided booting another gluster at sister-site too complex
  • Gluster snapshots
    • Implemented via changing gluster to use LVM thin provisioning
    • Snapshotting is closer to backup, but relies on the local filesystem storage
      • Makes roll-back in time possible, but it not truly a copy
      • Still needs to be copied off-site
      • Gluster snapshots are not first-class object - can't be easily exported
      • LVM thin provision adds non-significant overhead
        • undesirable for a globally used filesystem
    • Was implemented as LVM TP and ansible plays
      • Very complex ansible was difficult
      • Run from provisioner via cron questionable reliability
        • provision node dependencies and complexity
        • Brittle in face of errors
  • Direct backups of gluster raw-data via deployed sidecars in cluster
    • Cleaner and simpler design
      • Runs in-cluster, managed by kubernetes
        • add reliability
      • Fire and forget
        • No provisioner dependency

 

The implemented design - in-cluster backup sidecars and ansible provisioning scripts

  • Cluster-backup sidecar container (ndslabs/cluster-backup) deployed as kubernenetes replicated service at provision-time.
    • 1-1 gluster server:gluster backup sidecars
      • scalable - scales at run-time without intervention
      • resilient to single failures
    • configuration data from the gluster-server config and from backup-config
      • no change to existing codes
      • backup config sensitive
        • ssh credentials - stored in kubernetes secret
    • Backup algorithm:
      • mounts the xfs data, xfsdump live to remote storage independently
      • Dumps etcd and store remotely
      • Dump kubernetes (info) and store remotely
  • Ansible plays for deploy-time
    • inject configuration to cluster
    • start replication sidecar set
    • dump provisioner data and store
    • simple, easy to understand, maintainable
    • The playbooks:
      • backup-saved-and-sensitive-now.yml - dumps the provisioner saved/sensitive date to the backup store by an operator
      • deploy-cluster-backups.yml - deploys the cluster backup daemonset into a cluster, useful for adding backups to a cluster previously provisioned on earlier deploy-tools
      • setup-local-kubectl.yml - sets up kubectl to use directly from provisioner - for use to update a cluster that was provisioned on earlier deploy-tools.
    • New deploys:    include setup-local-kubectl and deploy-cluster-backups in the deployment role.

Restore Processes:

restore needs some advanced discussion to tease out details relating to scenarios.  There are intricacies in etcd and gluster which both rely on state that specifies IP addresses, volumes, etc.    It is unlikely that a fully identical system restore wil be done but more likely sub-set scenios.

  • filesystem data recovery
  • full system recovery
  • data/cluster migration
  • lost-data backup, etc.

Specific Restore Procedures

  • glfs
    • shut down glfs server pods
    • Restore can be done in-place from the backup container for xfsrestore access
      • arrange access to the file, either copying to local or via remote mount
      • cd /media/brick0
      • xfsrestore -f f <backupfile>
    • restore each brick state with xfsrestore
      • format the xfs filesystem
      • either copy intended backup file to local or run from remote ssh/cat to xfsrestore
      • Restore the gluster host-side gluster and backup state on the host glfs node from the host-state.tgz file

        • /var/lib/glusterd

        • /var/log/glusterfs

        • /etc/glusterfs

        • /media/brick0/brick

        • /var/lib/xfsdump

      • redeploy the gluster-server pods
        • glusterfs should recover and all nodes should come on line.
    • xfs admin information with xfsdump/restore is here: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/xfsbackuprestore.html
  • etcd
    • etdc is not intended to be restored - it relies on the 5 or 7 instances for repliction
    • An etcd backup can be applied to a new and fresh etc cluster
    • the procedure is in the CoreOS disaster recovery portion of the etcd admin guide:  https://coreos.com/etcd/docs/latest/admin_guide.html#disaster-recovery
    • if you want a subset of etcd config data, restore to a new etcd cluster then pull it out with etcdctl.
  • No labels