Related to  NDS-1027 - Getting issue details... STATUS

  1. Compute Engine > VM Instances > Create Instance
    1. 2 vCPU, 7.5 GB memory
    2. CoreOS Stable with standard boot disk
    3. Add 100GB persistent disk
  2. SSH into instance
    1. sudo mkfs -type ext4 /dev/sdb
    2. mkdir /ndslabs
    3. sudo mount /dev/sdb /ndslabs

    4. sudo systemctl stop docker

    5. mount --bind /ndslabs/docker/ /var/lib/docker

    6. Add /etc/systemd/system mount, if desired
  3. Start Workbench
    1. git clone https://github.com/nds-org/ndslabs-startup
    2. cd ndslabs-startup/
    3.  vi templates/config.yaml 
    4. Add ~/bin to PATH
      1. mkdir ~/bin
      2. export PATH=/home/you/bin:$PATH
      3. Edit bashrc
    5. kube up.sh
    6. ndslabs.sh up
  4. Add NBI mongo yaml
    1. kubectl create -f mongo.yaml --namespace=admin
  5. Create Spark cluster
    1. DataProc > Clusters > Create Cluster
    2. 3 notes, each with 2 vCPU adn 40GB primary disk
    3. Centos
    4. DataProc 1.1 (Spark 2.2)

      1. https://cloud.google.com/dataproc/docs/concepts/dataproc-versions

  6. Install livy
    1. SSH to edge

    2. apt-get install maven

    3. git clone https://github.com/cloudera/livy

    4. mvn package -D skipTests
    5. mkdir logs
    6. chmod a+w logs

    7. export SPARK_HOME=/usr/lib/spark

    8. export HADOOP_CONF_DIR=/etc/hadoop/conf

    9. bin/livy-server
  7. Start Zeppelin per  NDS-1027 - Getting issue details... STATUS

  • No labels