Original approach was to upgrade one version of play at a time and keep all functionality. This turned out incredible difficult given all the interdependencies between libraries and scala version. See old branch: https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/clowder/compare/commits?sourceBranch=refs%2Fheads%2F2.0&targetBranch=refs%2Fheads%2Fdevelop.
Currently working on this: Luigi Marini Rob Kooper Maxwell Burnette Todd Nicholson
New approach is as follows:
- Remove all unused and legacy code. This will be 2.0.
- Elasticsearch upgrade
- Pull request exists to upgrate 2.3.5 to 7.5
- Modify code to be compatible with newer versions of Playframework.
4.0+- Should jump straight to 2.7 or 2.6. We need to figure out which one.
- Update to Play 2.8, Update Scala to 2.12, keep Casbah because now it supports 2.12
- One branch we all work on to do this that will be broken for a while. We can make PRs but we can't run the code in progress.
- SEE BELOW FOR DETAILS
- Migrate authentication / authorization:
- New MongoDB implementation.
This will be 3.0+.- Replace Salat and Cashbah with Java Mongo Driver and keep the rest as is
- Split services in two layers.
- Top layer has the common code that calls the other services. For example when removing a dataset also remove files.
- Bottom layer is traits to do CRUD operations on resources and requires multiple implementations.
- How do tests factor in? can we start writing them before the refactoring? For example unit tests of services vs play scala tests of controllers actions.
- Enforce style as part of this process. Using IDE? We should look at what rules and style software to use.
PLUGINS
Here is a list of plugins that, as of 1/6/2020, nobody has created a pull request yet to turn into a service. Can any be deleted or ignored, and who is doing them in what order?
- MailerPlugin (it was not clear if this plugin actually gets used to send email to me - Todd.)
PolyglotPlugin (pull request created, 27 Jan 20)PostgresPlugin (removal added to remove geostreams pull request, 27 Jan 20)- RDFExporterPlugin
RDFExportService (implements plugin)- Since the RDF endpoints have been removed, can the RDF plugins be removed as well?
- Pull request created for removing these plugins (note added - 1/15/2020)
- SecureSocialTemplatesPlugin
SpaceCopyPlugin - Todd - will replace with value in conf. This plugin is simply enabled or not. (Pull request created)
SpaceSharingPlugin- Todd - will replace with value in conf, again, either enabled or not. (Pull request created)
StagingAreaPlugin- This plugin worked pretty much the same as SpaceCopy and SpaceSharing, so I replaced it with a value in the application.conf in the same pull request as those 2 plugins. (Todd, 1/7/2020).
- TempFilesPlugin
ToolManagerPlugin- Removed by Todd (pull request now open)
VersusPlugin- Beginning this week (todd, 22 Jan 2020)
PyClowder
There are several significant changes to PyClowder that have been discussed - would be nice to bundle them into a corresponding release update for 2.0. (essentially pyclowder 3.0)
- Move to a client-based structure entirely with standardized support for both key and user/pass, along with other improvements. Max has a branch from Spring 2019 with many of these implemented: https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/pyclowder/compare/diff?sourceBranch=refs%2Fheads%2Fclowder-clients-clean&targetRepoId=1013
- Improve support for extractor versioning
- Include extractor version in attached metadata (perhaps in agent section, also needs Clowder upgrades for this)
- Allow registration of multiple versions of same extractor and make sure API points to correct version
- Flags for metadata upload functions to increase flexibility
- overwrite metadata if from same name + version of extractor
- skip extraction of metadata from same name + extractor exists
- optional flag to ignore version matching?
- move simple extractor into pyclowder formally so it's easy to import
- move sample simple extractors somewhere else?
Play 2.6 Upgrade
Sbt migration: http://www.scala-sbt.org/0.13/docs/Migrating-from-sbt-012x.html
Play 2.8 Upgrade
- Luigi started a Play 2.6 branch in progress - Will merge this in our current 2.0
Library Update Status Oct. 2020
- Replacing SecureSocial with Play Silhouette (https://github.com/mohiva/play-silhouette-seed/tree/5.0.0)
- User model is broken
- Calls to Identity broken
- Change in Requests (related?) - changed action signatures
- Useful pages from discussion
- Mongo Salat
- Replacing DAO patterns, Luigi started work, Todd bugfixes, might be done.
- some IDE errors may also exist in master
- Updating common Play imports
- Configuration if any remaining
- Iteratee/Enumerator replacements
- Some stronger typing requirements where we have types inferred
- Play import changes (same code, different paths mostly