Uploaded image for project: 'National Data Service'
  1. National Data Service
  2. NDS-700

/var/glfs/global: Transport endpoint is not connected

XMLWordPrintableJSON

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • None
    • None
    • Infrastructure
    • None
    • NDS Sprint 18

      For some unknown reason, the NFS mount to /var/glfs/global can fail or become disconnected in some way.

      This has happened on both Labs Workbench Beta and my small-scale mltest cluster.

      The most obvious way to see this error is by simply running df:

      core@workbench-node1 ~ $ df
      		 
      df: /var/glfs/global: Transport endpoint is not connected
      		 
      Filesystem     1K-blocks     Used Available Use% Mounted on
      		 
      devtmpfs        32973900        0  32973900   0% /dev
      		 
      tmpfs           32990036        0  32990036   0% /dev/shm
      		 
      tmpfs           32990036     2272  32987764   1% /run
      		 
      tmpfs           32990036        0  32990036   0% /sys/fs/cgroup
      		 
      /dev/vda9       78853524  5140600  70376896   7% /
      		 
      /dev/vda4        1007760   619300    336444  65% /usr
      		 
      tmpfs           32990036        0  32990036   0% /media
      		 
      tmpfs           32990036        0  32990036   0% /tmp
      		 
      /dev/vda1         130798    72466     58332  56% /boot
      		 
      /dev/vda6         110576       64    101340   1% /usr/share/oem
      		 
      /dev/vdb       104806400 58902200  45904200  57% /var/lib/docker
      		 
      tmpfs            6598008        0   6598008   0% /run/user/500
      		 
      core@workbench-node1 ~ $

      The output from dmesg -T when this error occurs can be found below:

      core@mltest-node1 ~ $ dmesg -T
      		 
      [Tue Nov 29 09:26:31 2016] Linux version 4.7.3-coreos-r2 (jenkins@jenkins-os-executor-1.c.coreos-gce-testing.internal) (gcc version 4.9.3 (Gentoo Hardened 4.9.3 p1.5, pie-0.6.4) ) #1 SMP Tue Nov 1 01:38:43 UTC 2016
      		 
      [Tue Nov 29 09:26:31 2016] Command line: BOOT_IMAGE=/coreos/vmlinuz-b mount.usr=PARTUUID=e03dd35c-7c2d-4a47-b3fe-27f15780a57c rootflags=rw mount.usrflags=ro consoleblank=0 root=LABEL=ROOT console=ttyS0,115200n8 console=tty0 coreos.oem.id=openstack
      		 
      [Tue Nov 29 09:26:31 2016] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
      		 
      [Tue Nov 29 09:26:31 2016] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
      		 
      [Tue Nov 29 09:26:31 2016] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
      		 
      [Tue Nov 29 09:26:31 2016] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
      		 
      [Tue Nov 29 09:26:31 2016] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
      		 
              .           .           .           
      [Tue Nov 29 13:08:48 2016] overlayfs: upper fs needs to support d_type.
      		 
      [Tue Nov 29 13:08:48 2016] overlayfs: upper fs needs to support d_type.
      		 
      [Tue Nov 29 13:08:48 2016] overlayfs: upper fs needs to support d_type.
      		 
      [Tue Nov 29 13:09:07 2016] veth173c410: renamed from eth0
      		 
      [Tue Nov 29 13:09:07 2016] docker0: port 4(veth8300865) entered disabled state
      		 
      [Tue Nov 29 13:09:07 2016] docker0: port 4(veth8300865) entered blocking state
      		 
      [Tue Nov 29 13:09:07 2016] docker0: port 4(veth8300865) entered forwarding state
      		 
      [Tue Nov 29 13:09:07 2016] docker0: port 4(veth8300865) entered disabled state
      		 
      [Tue Nov 29 13:09:07 2016] device veth8300865 left promiscuous mode
      		 
      [Tue Nov 29 13:09:07 2016] docker0: port 4(veth8300865) entered disabled state
      		 
      [Tue Nov 29 13:09:18 2016] docker0: port 5(veth6d2c88f) entered disabled state
      		 
      [Tue Nov 29 13:09:18 2016] vethb1d0170: renamed from eth0
      		 
      [Tue Nov 29 13:09:18 2016] docker0: port 5(veth6d2c88f) entered blocking state
      		 
      [Tue Nov 29 13:09:18 2016] docker0: port 5(veth6d2c88f) entered forwarding state
      		 
      [Tue Nov 29 13:09:18 2016] docker0: port 5(veth6d2c88f) entered disabled state
      		 
      [Tue Nov 29 13:09:18 2016] device veth6d2c88f left promiscuous mode
      		 
      [Tue Nov 29 13:09:18 2016] docker0: port 5(veth6d2c88f) entered disabled state
      		 
      [Tue Nov 29 13:14:50 2016] blk_update_request: I/O error, dev vdb, sector 161315632
      		 
      [Tue Nov 29 13:14:50 2016] blk_update_request: I/O error, dev vdb, sector 161315624
      		 
      [Tue Nov 29 13:14:50 2016] Buffer I/O error on dev vdb, logical block 20164453, lost async page write
      		 
      [Tue Nov 29 13:14:50 2016] blk_update_request: I/O error, dev vdb, sector 104953929
      		 
      [Tue Nov 29 13:14:50 2016] blk_update_request: I/O error, dev vdb, sector 104953929
      		 
      [Tue Nov 29 13:14:50 2016] XFS (vdb): metadata I/O error: block 0x6417849 ("xlog_iodone") error 5 numblks 64
      		 
      [Tue Nov 29 13:14:50 2016] XFS (vdb): xfs_do_force_shutdown(0x2) called from line 1200 of file ../source/fs/xfs/xfs_log.c.  Return add
      		 
      [Tue Nov 29 13:14:50 2016] XFS (vdb): Log I/O Error Detected.  Shutting down filesystem
      		 
      [Tue Nov 29 13:14:50 2016] XFS (vdb): Please umount the filesystem and rectify the problem(s)
      		 
      [Tue Nov 29 13:14:50 2016] XFS (vdb): xfs_log_force: error -5 returned.
      		 
      [Tue Nov 29 13:14:50 2016] blk_update_request: I/O error, dev vdb, sector 104953931
      		 
      [Tue Nov 29 13:14:50 2016] blk_update_request: I/O error, dev vdb, sector 104953934
      		 
      [Tue Nov 29 13:14:50 2016] blk_update_request: I/O error, dev vdb, sector 104953937
      		 
      [Tue Nov 29 13:14:50 2016] blk_update_request: I/O error, dev vdb, sector 104953940
      		 
      [Tue Nov 29 13:14:50 2016] XFS (vdb): metadata I/O error: block 0x641784b ("xlog_iodone") error 5 numblks 64
      		 
      [Tue Nov 29 13:14:50 2016] XFS (vdb): xfs_do_force_shutdown(0x2) called from line 1200 of file ../source/fs/xfs/xfs_log.c.  Return add
      		 
      [Tue Nov 29 13:14:50 2016] XFS (vdb): metadata I/O error: block 0x641784e ("xlog_iodone") error 5 numblks 64
      		 
      [Tue Nov 29 13:14:50 2016] XFS (vdb): xfs_do_force_shutdown(0x2) called from line 1200 of file ../source/fs/xfs/xfs_log.c.  Return add
      		 
      [Tue Nov 29 13:14:50 2016] XFS (vdb): metadata I/O error: block 0x6417851 ("xlog_iodone") error 5 numblks 64
      		 
      [Tue Nov 29 13:14:50 2016] XFS (vdb): xfs_do_force_shutdown(0x2) called from line 1200 of file ../source/fs/xfs/xfs_log.c.  Return add
      		 
      [Tue Nov 29 13:14:50 2016] XFS (vdb): metadata I/O error: block 0x6417854 ("xlog_iodone") error 5 numblks 64
      		 
      [Tue Nov 29 13:14:50 2016] XFS (vdb): xfs_do_force_shutdown(0x2) called from line 1200 of file ../source/fs/xfs/xfs_log.c.  Return add
      		 
      [Tue Nov 29 13:14:50 2016] XFS (vdb): xfs_log_force: error -5 returned.
      		 
      [Tue Nov 29 13:14:50 2016] blk_update_request: I/O error, dev vdb, sector 161315640
      		 
      [Tue Nov 29 13:14:50 2016] blk_update_request: I/O error, dev vdb, sector 161315648
      		 
      [Tue Nov 29 13:14:50 2016] Buffer I/O error on dev vdb, logical block 20164454, lost async page write
      		 
      [Tue Nov 29 13:14:50 2016] Buffer I/O error on dev vdb, logical block 20164455, lost async page write
      		 
      [Tue Nov 29 13:14:50 2016] Buffer I/O error on dev vdb, logical block 20164456, lost async page write
      		 
      [Tue Nov 29 13:14:50 2016] Buffer I/O error on dev vdb, logical block 289747, lost async page write
      		 
      [Tue Nov 29 13:14:50 2016] Buffer I/O error on dev vdb, logical block 291269, lost async page write
      		 
      [Tue Nov 29 13:15:13 2016] XFS (vdb): xfs_log_force: error -5 returned.
      		 
      [Tue Nov 29 13:15:43 2016] XFS (vdb): xfs_log_force: error -5 returned.
      		 
      [Tue Nov 29 13:16:13 2016] XFS (vdb): xfs_log_force: error -5 returned.
      		 
      [Tue Nov 29 13:16:43 2016] XFS (vdb): xfs_log_force: error -5 returned.
      		 
      [Tue Nov 29 13:17:13 2016] XFS (vdb): xfs_log_force: error -5 returned.
      		 
      [Tue Nov 29 13:17:43 2016] XFS (vdb): xfs_log_force: error -5 returned.
      		 
              .           .           .

      More investigation is needed into both the cause of this issue, as well as the circumstances under which it occurs.

              raila David Raila
              lambert8 Sara Lambert
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved:

                  Estimated:
                  Original Estimate - 4 hours
                  4h
                  Remaining:
                  Remaining Estimate - 4 hours
                  4h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified