Original approach was to upgrade one version of play at a time and keep all functionality. This turned out incredible difficult given all the interdependencies between libraries and scala version. See old branch: https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/clowder/compare/commits?sourceBranch=refs%2Fheads%2F2.0&targetBranch=refs%2Fheads%2Fdevelop.

Currently working on this: Luigi Marini Rob Kooper Maxwell Burnette Todd Nicholson

New approach is as follows:

•  Remove all unused and legacy code. This will be 2.0.
  • XML RDF export
  • Rabbitmq monitoring using manager API (Maxwell Burnette started on this)
  • Geostreams API (Todd Nicholson will do this ) - branch https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/clowder/compare/commits?sourceBranch=refs%2Fheads%2Frefactor%2F2.0-remove-geostreams
  • Deprecated endpoints (Maxwell working on this)
  • Every plugin should become a Guice Trait + Implementation
    • For services that are not implemented, provide an no-op implementation
    • Enabled/disabled, is it in the config? Is it a common trait and the ability to provide implementation class in config file?
  • ??? (Add by end of this week Nov. 1st)
•  Elasticsearch upgrade
  • Pull request exists to upgrate 2.3.5 to 7.5 
•  Modify code to be compatible with newer versions of Playframework. 4.0+
  • Should jump straight to 2.7 or 2.6. We need to figure out which one. 
    • Update to Play 2.8, Update Scala to 2.12, keep Casbah because now it supports 2.12
      • One branch we all work on to do this that will be broken for a while. We can make PRs but we can't run the code in progress.
      • SEE BELOW FOR DETAILS
  • Migrate authentication / authorization:
    
    • https://github.com/mohiva/play-silhouette
•  New MongoDB implementation. This will be 3.0+.
  • Replace Salat and Cashbah with Java Mongo Driver and keep the rest as is
    • Mongo drivers
      • Looks like official Scala driver is being kept up to date and is based on the latest Java Mongo Async Driver
        •  https://github.com/mongodb/mongo-scala-driver
          • requiresscalaVersions=("2.11.12", "2.12.8", "2.13.0")
        • Clowder - running Play 2.2.6 on Scala 2.10
          
          • Play 2.3 and 2.4 supported both Scala 2.10 and 2.11.
          • Play 2.5 has dropped support for Scala 2.10 and now only supports Scala 2.11.
          • 2.3 is first version of Play to support Scala 2.11
          • Play 2.8 support Scala 2.12
        • Casbah
          • compiled for Scala 2.11.8, 2.10.6, 2.12.1
        • Salat 
          • https://github.com/salat/salat
          • Scala 2.10, 2.11 and 2.12. Based on Casbah 3.1.1
      • http://mongodb.github.io/mongo-java-driver/ (this might be our safest bet)
      • http://mongodb.github.io/mongo-java-driver-reactivestreams/
      • https://mongodb.github.io/mongo-scala-driver/
  • Split services in two layers.
    • Top layer has the common code that calls the other services. For example when removing a dataset also remove files.
    • Bottom layer is traits to do CRUD operations on resources and requires multiple implementations.
•  How do tests factor in? can we start writing them before the refactoring? For example unit tests of services vs play scala tests of controllers actions.
•  Enforce style as part of this process. Using IDE? We should look at what rules and style software to use.
  • http://www.scalastyle.org/
  • Pick our preferred rules

PLUGINS

Here is a list of plugins that, as of 1/6/2020, nobody has created a pull request yet to turn into a service. Can any be deleted or ignored, and who is doing them in what order? 

• MailerPlugin (it was not clear if this plugin actually gets used to send email to me - Todd.)
• PolyglotPlugin (pull request created, 27 Jan 20)
• _{PostgresPlugin} (removal added to remove geostreams pull request, 27 Jan 20)
• RDFExporterPlugin 
• RDFExportService (implements plugin)
  • Since the RDF endpoints have been removed, can the RDF plugins be removed as well? 
  • Pull request created for removing these plugins (note added - 1/15/2020)
• SecureSocialTemplatesPlugin
• SpaceCopyPlugin 
  • Todd - will replace with value in conf. This plugin is simply enabled or not. (Pull request created)
• SpaceSharingPlugin
  • Todd - will replace with value in conf, again, either enabled or not. (Pull request created)
• StagingAreaPlugin
  • This plugin worked pretty much the same as SpaceCopy and SpaceSharing, so I replaced it with a value in the application.conf in the same pull request as those 2 plugins. (Todd, 1/7/2020). 
• TempFilesPlugin
• ToolManagerPlugin
  • Removed by Todd (pull request now open)
• VersusPlugin
  • Beginning this week (todd, 22 Jan 2020)

PyClowder

There are several significant changes to PyClowder that have been discussed - would be nice to bundle them into a corresponding release update for 2.0. (essentially pyclowder 3.0)

• Move to a client-based structure entirely with standardized support for both key and user/pass, along with other improvements. Max has a branch from Spring 2019 with many of these implemented: https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/pyclowder/compare/diff?sourceBranch=refs%2Fheads%2Fclowder-clients-clean&targetRepoId=1013
• Improve support for extractor versioning
  • Include extractor version in attached metadata (perhaps in agent section, also needs Clowder upgrades for this)
  • Allow registration of multiple versions of same extractor and make sure API points to correct version
• Flags for metadata upload functions to increase flexibility
  • overwrite metadata if from same name + version of extractor
  • skip extraction of metadata from same name + extractor exists
  • optional flag to ignore version matching?
• move simple extractor into pyclowder formally so it's easy to import
  • move sample simple extractors somewhere else?

Play 2.6 Upgrade

• Upgrade Play 2.2.0 to 2.2.6
• https://www.playframework.com/documentation/2.5.x/Migration23
• https://www.playframework.com/documentation/2.5.x/Migration24
  • http://www.silhouette.rocks/ - Secure Social replacement
    • example: https://github.com/adrianhurt/play-silhouette-credentials-seed
• https://www.playframework.com/documentation/2.5.x/Migration25
• https://www.playframework.com/documentation/2.6.x/Migration26

Sbt migration: http://www.scala-sbt.org/0.13/docs/Migrating-from-sbt-012x.html

Play 2.8 Upgrade

• Luigi started a Play 2.6 branch in progress - Will merge this in our current 2.0

Library Update Status Oct. 2020

• Replacing SecureSocial with Play Silhouette (https://github.com/mohiva/play-silhouette-seed/tree/5.0.0)
  • User model is broken
  • Calls to Identity broken
  • Change in Requests (related?) - changed action signatures
  • Useful pages from discussion
    • https://www.silhouette.rocks/docs/environment
      • Create an environment with our own User and Authenticator
    • https://www.silhouette.rocks/docs/endpoints
      • Hopefully we can use their actions primarily.
    • https://www.silhouette.rocks/docs/authenticator (& Authorization, Identity)
      • Use this for custom permission WithProvider... we can do our own permission check on the action
      • SessionAuthenticator to start
    • providers
      • CI Logon authenticator necessary?
      • API key provider
• Mongo Salat
  • Replacing DAO patterns, Luigi started work, Todd bugfixes, might be done.
  • some IDE errors may also exist in master
• Updating common Play imports
  • Configuration if any remaining
  • Iteratee/Enumerator replacements
  • Some stronger typing requirements where we have types inferred
  • Play import changes (same code, different paths mostly