Versions Compared

Old Version 9

changes.mady.by.user Sara Lambert

Saved on

compared with

New Version Current

changes.mady.by.user Sara Lambert

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

All documentation pertaining to how developers can contribute to NDS Labs.

Table of Contents

New to NDS Labs?

Start here: New Developer Workflow

Table of Contents

...

Develop Workflows

...

  • Git Workflows: Forking workflow with feature branches
    1. Fork repo (if applicable)
      • Press "Fork" in GitHub UI
    2. Clone repo to make changes locally (if applicable)
    3. Ensure correct branch and sync with upstream before making additional changes
      • git checkout master
      • git pull upstream master
    4. Create a branch named after the Story (for example 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-174
      )
      • git checkout -b NDS-174
    5. Make any necessary modifications locally on your branch
    6. Stage any modified files for commit
      • git add path/to/modified/file.ext
    7. Commit any modifications to your local branch with a comment
      • git commit -m "A comment about this commit"
    8. Push any local commits back up to your remote branch (your forked repo)
      • git push origin NDS-174
    9. When you are satisfied with your changesset of commits, create a Pull Request (PR) to view the diff
      • Press "Pull Request" in GitHub UI
      • Be sure to select the correct base and compare branches
        • Select nds-org/ndslabs as the base fork
        • Select master as the base branch
        • Select your personal fork (USERNAME/ndslabs) as the head fork
        • Select your personal Story branch as the compare branch
      • Scroll down and click on the "Files Changed" tab to briefly review your own Pull Request
        • Ensure that all changes made on this branch were intentional
        • If you are unsure about any specific code segments, comment in-line on the PR to ask for clarification
        • If you are unsure about any general concepts changed or introduced, comment in the section at the bottom of the PR
      • Name your Pull Request after the Story / branch (i.e. "NDS-174: User can access console of running service via CLI") 
      • Enter a short description of any modifications, additions, or removals from the codebase
        • If applicable, include a Test Case that the reviewer should run before merging the Pull Request
      • Click "Create Pull Request"
  • Docker Workflows: Upload Push any necessary test images to Docker Hub
    1. Build test image
      • docker build -t ndslabs/apiserver:dev .
    2. Tag test image with Story id (i.e. NDS-174)
      • docker tag ndslabs/apiserver:dev ndslabs/apiserver:NDS-174
    3. Push test image to Docker Hub
      • docker push ndslabs/apiserver:NDS-174
  • Kubernetes Workflows: Sometimes used in testing new services or the API server

JIRA Workflows

To handle issue and project tracking we use JIRA, which currently offers several different Issue Types when creating new tickets.

The expected usage of each Issue Type is outlined briefly in the workflows below:

Addition Workflows

Issue Types Used:

  • Wish / New Feature: a high-level non-technical description of desired business logic
  • Requirements: a "discussion" ticket describing a new feature that needs more of its technical description fleshed out
  • Epic: a high-level technical description of desired software functionality or infrastructure containing multiple Story tickets
    • Note: Epic issues do not get explicitly added to Sprint
  • Story: a use case describing an example usage of a small piece of newly desired functionality

Possible Stages:

  1. A New Feature or Wish ticket is filed describing the new feature at a high-level without mandating any particular technical specifications
  2. A Requirements ticket is created if we do not have enough information to break the feature down into small pieces of technical work
  3. A Story ticket is filed from the information resulting from the Requirements discussion
    1. If multiple Story tickets encompass the work needed, these tickets are grouped under an over-arching Epic ticket

New Feature / Wish Workflow

When a Wish or New Feature ticket is in the Active Sprint:

  1. The ticket is marked IN PROGRESS and assigned to a developer, who conducts the meeting
  2. The feature is broken down into a digestable number of enumerated use cases
  3. If one or more use cases require more detail, a Requirements ticket is filed
  4. The use cases are filed as Story tickets and associated to an Epic
  5. The resulting Story tickets are then discussed at the next Sprint Planning meeting

Deliverables:

  • New tickets describing the next steps necessary to enable the feature described

Requirements Workflow

When a Requirements ticket is in the Active Sprint:

  1. The ticket is marked IN PROGRESS and assigned to a developer, who conducts the meeting
  2. A meeting is created in Outlook to contact interested parties (i.e. NDS Labs Dev team, Nebula team, other NDS-affiliated software teams, etc.)
    1. The requirements are discussed with the development team and any interested parties
    2. Any information resulting from the discussion is filed into a Confluence wiki page
    3. The information from the Confluence page generates use cases
    4. If applicable, a new Epic is created to encompass the use cases presented
  3. The use cases are filed as Story tickets and associated to an Epic
  4. The ticket is marked IN REVIEW and assigned to another team member for review
    1. The reviewer may make any changes or comments that they desire and discuss with the team
  5. The ticket is marked RESOLVED or CLOSED
  6. The resulting Epic / Story tickets are then discussed at the next Sprint Planning meeting

Deliverables:

  • New Epic / Story tickets describing the steps necessary to enable the use cases described

Story Workflow

When a Story ticket is in the Active Sprint:

  1. The ticket is marked IN PROGRESS and assigned to a developer (referred to hereafter as "the developer")
  2. The developer does the work necessary to enable the use case described in the ticket
    1. Follow the general development workflows defined above
  3. Comment on the Story with links to any deliverables that should be reviewed / tested
  4. The ticket is marked IN REVIEW and assigned to a tester (referred to hereafter as "the tester")
  5. The tester reviews the deliverables of the Story:
    1. Review any related Pull Requests
    2. Review any Test Cases / Documentation provided
    3. Review any new tickets resulting from the work done
    4. Pull and run any new Docker images against the Test Cases provided
  6. The tester selects Review Accepted and the ticket is marked as RESOLVED
  7. The tester merges any outstanding Pull Requests related to this ticket
  8. The developer switches back to their master branch and syncs with the new changes
  9. If applicable, the developer builds and pushes a new "latest" Docker image for the API / UI incorporating the new changes
  10. The developer selects CLOSE TICKET and the ticket is marked as CLOSED

Deliverables:

  • Pull Request(s) to GitHub
  • New Image(s) / Tag(s) in Docker Hub
  • Test Case(s) demonstrating the use case fulfilled by the story
  • Documentation describing the technical aspects of the solution

Modification Workflows

Improvement Workflow

Issue Types Used:

  • Improvement: a suggestion to utilize new techniques or technologies to improve overall performance or maintainability

Bug Workflow

Issue Types Used:

  • Bug: a previously completed use case or edge case that is malfunctioning according to its defined behavior

Miscellaneous Workflows

Task Workflow

Issue Types Used:

  • Task: work that is not driven by a new use case containing zero or more Sub-Issue tickets (i.e. collaboration with other teams)
    • Technical Task: a small piece of technical work that is not driven by a new use case
    • Sub-Task: a small piece of outreach or non-technical work / discussion that is not driven by a new use case

Used to Track:

  1. events requiring special attention (i.e. hackathon, developer tutorial, etc.)

Comment Workflow

Issue Types Used:

  • Comment: track miscellaneous information / feedback / requests that do not match other issue types:

Used to Track:

  1. new sites / groups / contacts wishing to utilize the NDS Labs platform (i.e. Odum, TACC, SDSC, etc.)
  2. similar technologies that we might look at for reference (i.e. JujuCharms, ProfitBricks, etc.)
  3. new or existing technologies that might be leveraged to further NDS Labs

Request Workflow

Issue Types Used:

  • Processing Request: track the creation of new entities in production instance of NDS Labs

Used To Track:

An Example

https://github.com/bodom0015/developer-workflow

More Detail

Children Display
alltrue

Release Workflows

New Unstable "Test" Release

Prerequisites

  • Pull request has been created containing the changes to be reviewed / tested
  • Ensure that associated JIRA ticket contains a test case
  • Ensure that your current code passes the test case that you have written
  • Ensure documentation in Confluence is up-to-date
  • Checkout your feature branch
    • git checkout master
  • Sync with upstream
    • git pull upstream master
    • git push origin master
  • Update any relevant documentation in GitHub

Developer's Process (Semi-automatic)

  1. "Start Progress" on one of your assigned tickets (assign a new one if you have none assigned)
  2. If you haven't already, fork the upstream repository (you will only need to do this once per repository)
    • Set up an automatic build of your new fork on DockerHub
    • Configure the build to build all new branches from GitHub (choose "Branch" and leave the branch name blank)
    • Push these new branch builds to a Docker image tag of the same name (simply leave the tag name blank)
    • Once saved, 
  3. Clone your fork onto your local machine
  4. Create / switch to a development branch (named after one the JIRA ticket associated with the work being done, i.e. NDS-XXX)
  5. Make any necessary changes to fulfill the JIRA ticket
  6. Commit all associated changes and push them to GitHub
    • Any new changes pushed to any branch on your GitHub fork will be automatically built into an image of the same name on DockerHub
  7. Mark ticket as "In Review" and assign to an available Tester
  8. Wait for the ticket to be assigned back to you
  9. Review the Tester's results
    • If the Tester encountered problems, choose Review Rejected go back to #5 and address them
    • If the Tester submitted comments or feedback, do your best to address their concerns or comment back to come to consensus
    • If both Developer and Tester

Tester's Process (Manual)

  • Ensure that all associated auto-build images have completed their builds before beginning testing
    • Links to these images should be provided with the test case.
  • Run through the test case described in the associated ticket(s)
    • A test case should be provided in the comments of each ticket, where appropriate. If it is not, send it back to the Developer.
    • The test case should include success and / or failure criteria. If it does not, send it back to the Developer.
  • Update the associated JIRA tickets:
    • Briefly include the results of your testing
    • Be sure to leave feedback for the developer if you need them to take action
    • If something went wrong or the Tester still has questions, choose Review Rejected, assign it back to the Developer, and wait for a reply or the ticket to comeback to you
    • If all test cases pass according to the standards set in the ticket and its comments, choose Review Accepted and assign it back to the Developer
  • After all tickets in the associated JIRA tickets are marked as Resolved, merge the Pull Request into the master branch on the nds-org GitHub
    • Merging any PRs to upstream master will automatically trigger a build of "latest" on DockerHub (see below)

New Unstable "Latest" Release

Prerequisites

  • Any related PRs have been merged to master
  • Ensure that smoke test passes
  • Ensure documentation in Confluence is up-to-date
  • Checkout your master branch
    • git checkout master
  • Sync with upstream
    • git pull upstream master
    • git push origin master
  • Update ALL documentation in GitHub

Process (Automatic)

  • New "latest" Docker images are automatically built from the upstream master branch on GitHub.
  • All new changes that make it into master on GitHub will automatically trigger a build on DockerHub

Official Tagged Version Release (Stable)

Prerequisites

  • Ensure that all tests pass
  • Ensure documentation in Confluence is up-to-date
  • Checkout your master branch
    • git checkout master
  • Sync with upstream
    • git pull upstream master
    • git push origin master
  • Update ALL documentation in GitHub

Process (Semi-automatic)

  1. For each repo: branch off of develop to create a release branch
    • ndslabs and workbench-helm-chart

OR

    • kubeadm-bootstrap and kubeadm-terraform
  2. Roll versions forward
    1. ndslabs
      1. gui/swagger.yaml: NDS Labs swagger API spec version number
      2. apiserver/build.sh: NDS Labs API Server Docker image version tag
      3. apiserver/cmd/clientVersion.go: NDS Labs CLI version number / build date
      4. gui/Dockerfile: NDS Labs UI / webserver Docker image version tag
      5. gui/package.json: NDS Labs UI  / webserver NPM package version number
      6. gui/ConfigModule.js: NDS Labs UI Angular app build version number
    2. workbench-helm-chart
      1. values.yaml: NDS Labs API Server + UI Docker image version tags
    3. kubeadm-bootstrap
      1. install-kubeadm.bash: Kubernetes / Docker version numbers
      2. init-master.bash: Helm version numbers
    4. kubeadm-terraform
      1. kubeadm-bootstrap git release/tag: assets/bootstrap.sh
  3. PR from release branch to master
    1. Thorough testing, then more testing, then merge to master
  4. Tag master with new version number (freshly tested and stable)
    1. Any new merges and tags will trigger new Docker images to be built
  5. Wait for newly-tagged resources to automatically finish building and pushing Docker images
    1. Run a quick smoke test with newly-tagged resources
    2. Fix any last-minute errors directly on master and recreate release
  6. Backport any missing changes from release-x.x.x into develop
    1. This should include, at the very least, a commit from the release branch that rolls forward to new version numbers
    2. git checkout develop && git pull origin master && git push origin develop? Why does this not work with a PR...
Legacy Process
  1. Regenerate Swagger API / Client from spec (this can be skipped if the spec has not changed)
    • apiserver/???: generated Go swagger server
    • gui/js/app/shared/api.js: generated AngularJS swagger client 
  2. Roll forward version numbers in ndslabs-deploy-tools and ensure that all values match the version number you are about to create:
    • roles/cluster-backup/defaults/main.yml
    • roles/ndslabs-api-gui/defaults/main.yml
    • roles/k8s-nagios-nrpe/defaults/main.yml
    • roles/k8-glfs-server-pods/defaults/main.yml
    • roles/k8-glfs-client-set/defaults/main.yml
  3. Create a new tag from master in GitHub for the new version (i.e. 1.0.0, 1.0.1, etc):
    1. Repositories should be tagged in the following order when possible:
      1. ndslabs (API server / REST API / CLI / UI)
      2. ndslabs-specs (service specs - starting with 1.2.0, this can be versioned separately from Workbench, but it should be noted upon a new release which version of Workbench the specs release will target)
      3. workbench-helm-chart (Helm deployment to Kubernetes)
      4. gluster (global file system - deprecated, no longer used)
      5. cluster-backup (cron job for backing up glfs / etcd / kubectl dump - starting with 1.2.0, this can be versioned separately from Workbench)
      6. ndslabs-nrpe (nagios monitoring - these can now be versioned separately from the rest of Labs Workbench)
      7. ndslabs-startup (dev-cluster startup - deprecated, no longer used)
      8. ndslabs-deploy-tools (ansible scripts - deprecated, no longer used)
      9. kubeadm-bootstrap (kubernetes deployment scripts - versioned separately, but it should be noted upon a new release which version of Workbench the release will target)
      10. kubeadm-terraform (terraform deployment procedure - versioned separately, but it should be noted upon a new release which version of Workbench the release will target)
      11. ndslabs-devenvs (developer environments - these can now be versioned separately from the rest of Labs Workbench)
        • NOTE: ndslabs-devenvs contains a large number of cascading images that will quickly fill up the build queue, that's why we do it last
    2. New versioned Docker images are automatically built from the upstream tags created on GitHub.
    3. All new tags that are created will trigger a build
  4. Roll forward version numbers in source and ensure that all values match on upstream master on GitHub:
    • Swagger API
      • apis/swagger-spec/ndslabs.yaml: NDS Labs swagger API spec version number (deprecated - use the file below instead)
      • gui/swagger.yaml: NDS Labs swagger API spec version number
    • API Server:
      • apiserver/build.sh: NDS Labs API Server Docker image version tag
      • apiserver/version.go: NDS Labs API / Server version number file no longer exists
    • CLI Client:
      • apictl/build.sh: NDS Labs CLI version number file no longer exists
      • apictl/cmd/clientVersion.go: NDS Labs CLI / API version number file no longer exists
    • UI Client:
      • gui/Dockerfile: NDS Labs UI / webserver Docker image version tag
      • gui/package.json: NDS Labs UI  / webserver NPM package version number
      • gui/bower.json: NDS Labs UI Angular app Bower package version number file no longer exists
      • gui/ConfigModule.js: NDS Labs UI Angular app build version number
  5. projects (via Account Creation Workflow)
  6. service specs (via Pull Requests made to ndslabs-specs)