Page tree
Skip to end of metadata
Go to start of metadata

Introduction

Like other "fiddle" type environments such as JSfiddle, ScalaFiddle and Python Fiddle, BDFiddle provides an easy way for non-programmers to generate and play around with metadata and code "snippets" needed to generate metadata or convert/extract data from files they submit to Brown Dog. At its core, Brown Dog provides two kinds of transformation services, namely extraction and conversion. Extraction refers to the process of automatically generating specific metadata from a file. For example, detecting the number of faces in a photograph. On the other hand, conversion services deal with changing a file from one type to another (e.g. converting an MP4 video file to AVI format). There are multiple modes in which you can use BDFiddle. For example, for a regular user there is the "Automatic" (or basic) mode in which all available transformations (conversions or extractions) are applied to the submitted input file. For a more experienced user, there is the "Manual" (or advanced) mode in which the user can pick and choose specific transformations that they want to apply to the input file. BDFiddle is mainly meant to be a tool for exploring Brown Dog by playing around with its transformation services and hence currently we do not support uploading files that are larger than 5 MB through BDFiddle. If you need to test with larger files, there is a "Big Data" option in BDFiddle that will generate code snippets in multiple languages (currently shell script, native Python, and Brown Dog Python library are supported), which you can run at your end for local processing of files and uses Docker containers to move compute to data. You can also generate workflows using BDFiddle.

Navigation

Navigate to the Brown Dog website ans sign up:

https://browndog.ncsa.illinois.edu

Click on the "BD Fiddle" button on the left.

or you can navigate directly using the URL:

https://browndog.ncsa.illinois.edu/bdfiddle/

Login with your Brown Dog username and password.

You will be taken to the BDFiddle page itself.  This is what you will see. The whole page is setup to assist the user in:

  1. Selecting a file
  2. Selecting a transformation
  3. Receiving the output
  4. Accessing the code snippets

1. Select a File

You can choose from a file on your computer via the "Browse" button or select URL to choose a remote file.  The Examples link at the top provides a few example files to help you get going quickly.  Click on the "Get Transformations" button.

2. Select Transformation

Based on the type of the input file, a list of available conversions and extractions for that file are displayed.  Select the type of transformation needed from either of the scroll lists.  If you are using "Manual" mode of operation, you will still need to do the steps mentioned above, except that you can be more specific about the transformation that you want to perform, specifically what individual applications should be used.  Press "Submit" button to initiate the transformation.

Associated Options

  • Email Output: To receive the output via email, please select this checkbox and the output will be sent to the email address associated with your Brown Dog account. Currently this feature is available only for conversions.
  • Big Data: To obtain code snippets for processing large files locally without sending them to remote services, please select this checkbox. These snippets use Docker and it needs to be installed in your machine.
  • Generate Workflow: To generate workflows for your transformations using DataWolf (a web based scientific workflow management system associated), please select this checkbox.

3. Receive Output

If a conversion was selected - the output box will provide the URL of the file after converting to the desired format.  If an extraction was selected - the output box will display the extracted metadata.  The below example displays metadata in the output box generated from a JPG file after extraction.

4. Obtain Code Snippets

BDFiddle also generates code snippets, which can be used by the user to recreate the same transformation from within their applications. Currently we provide code snippets in Python, R, MATLAB, bash, and JavaScript using libraries for Brown Dog.  For each language there are four possible ways to interact with Brown Dog on a particular file or dataset: convert a file from a URL, convert a local file after uploading, extract from a file from a URL, and extract from a local file after uploading.  The appropriate snippet is generated according to the options you select from within BDFiddle.  Some example snippets are show below:

  • Terminal - shell script that can be run from the command line:

  • Python - snippet that uses bd.py (Brown Dog python library):

  • Native Python - snippet in native Python:

  • MATLAB - code to utilize in a MATLAB program to re-run the transformation:

  • R - code necessary to re-run the transformation in R language:

Jupyter

In response to the popularity of Jupyter notebooks and the ease of combining runable code and documentation in one interface that Jupyter provides, Jupyter functionalities have been added to BDFiddle.  In addition to copying the code snippet, a user can download a Jupyter notebook that contains the runnable code that is ready to implement locally (jupyter must be installed) or the user can simply open the notebook as an anonymous user on an NCSA multi-user service that utilizes tmpnb.  Jupyter templates are JSON files that contain both the raw code (runnable) and markdown (documentation).  Where Jupyter is available, two additional buttons appear below the code snippet.

The buttons enable a user to:

  • Download the Jupyter notebook - to use in a local installation of Jupyter - The downloaded file can be immediately opened in Jupyter by typing “jupyter notebook” in the directory where the file is located.
  • Open the Jupyter Notebook - the input file and the Jupyter notebook file are posted to the remote NCSA server where it can be accessed. 

When the files are posted to the server, a notebook container is allotted to the new user in which they can edit and run the Jupyter notebook.

See Jupyter documentation for specific usage of Jupyter notebooks:

https://jupyter.readthedocs.io/en/latest/

Report a Problem

Log in and click "Report Problem" in the Navigation bar then you will see the form below:

Fill in the form with your name and details of the problem. Screenshots or any file can be uploaded in the attachment.  If the report is sent successfully, you will see the message as follows:

 

  • No labels