You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 37 Next »

The Brown Dog Command Line Interface (BD-CLI) provides users with an easy to use interface to Brown Dog services. It talks to various components of Brown Dog through its API. It can be used to do operations like file conversions and extractions. Please see the README for current instructions.

Installation

You can install BD-CLI from the PyPi repository by following these steps:

  1. Create a Python virtual environment in your computer using virtualenv, virtualenvwrapper or any other tool.
  2. Activate the created Python virtual environment
  3. Run the following command: 

    pip install bdcli

  4. This will install bdcli and all the dependent packages and you should see a new application bd from your terminal.

Working with Terminal

This section is intended only to those users who are working in a terminal for the very first time or are not very experienced using it. Others, please free to skip this section.

TODO: Write content for this section

Example Transformations

Extraction

In the context of Brown Dog, extraction refers to the process of extracting relevant information from one or more data files. For example, finding out the location of people's faces in an image.

You can find an example of the extraction command below:

You can then read the contents of ncsa.json:

ncsa.json 
examples $ cat ncsa.json | jq .
{
  "metadata.jsonld": [
    {
      "content": {
        "Human Preference Extractor": {
          "Definitions": {
            "Human Preference": "A Computer Vision model that uses the spectral data of an image to get a human preference value ranging from 1 to 5.",
            "Green Index": "The green index is the estimated percentage of green pixels within an image."
          },
          "Data": {
            "Green Index": "2.3"
          }
        }
      },
      "created_at": "Tue Jun 20 10:34:06 CDT 2017",
      "agent": {
        "extractor_id": "http://clowder.ncsa.illinois.edu/extractors/deprecatedapi",
        "@type": "cat:extractor",
        "name": "http://clowder.ncsa.illinois.edu/extractors/deprecatedapi"
      }
    },
    {
      "content": {
        "y1": 288,
        "y0": 88,
        "x0": 920,
        "x1": 1220
      },
      "@context": [
        "https://clowder.ncsa.illinois.edu/contexts/metadata.jsonld",
        {
          "y1": "http://clowder.ncsa.illinois.edu/gi_detector#y1",
          "y0": "http://clowder.ncsa.illinois.edu/gi_detector#y0",
          "x0": "http://clowder.ncsa.illinois.edu/gi_detector#x0",
          "x1": "http://clowder.ncsa.illinois.edu/gi_detector#x1"
        }
      ],
      "created_at": "Tue Jun 20 10:33:54 CDT 2017",
      "agent": {
        "extractor_id": "https://clowder.ncsa.illinois.edu/clowder/api/extractors/gi_detector",
        "@type": "cat:extractor",
        "name": "https://clowder.ncsa.illinois.edu/clowder/api/extractors/gi_detector"
      }
    },
    {
      "content": {
        "ocr_text": "WEB BROWSER MOSAIC THE FIRST POPULAR GRAPHICAL BROWSER FOR THE WORLD WIDE WEB WAS CREATED BY MARC ANDREESSEN AND ERIC BINA AT THE NATIONAL CENTER FOR COMPUTING APPLICATIONS NCSA UPON ITS 1993 RELEASE To THE PUBLIC MOSAIC GAVE INTERNET USERS EASY ACCESS TO MULTIMEDIA SOURCES OF INFORMATION WEB BROWSERS HAVE TRANSFORMED THE EXCHANGE OP INFORMATION UNIVERSITY OF ILLINOIS"
      },
      "@context": [
        "https://clowder.ncsa.illinois.edu/contexts/metadata.jsonld",
        {
          "ocr_text": "http://clowder.ncsa.illinois.edu/ncsa.image.ocr#ocr_text"
        }
      ],
      "created_at": "Tue Jun 20 10:33:50 CDT 2017",
      "agent": {
        "extractor_id": "https://clowder.ncsa.illinois.edu/clowder/api/extractors/ncsa.image.ocr",
        "@type": "cat:extractor",
        "name": "https://clowder.ncsa.illinois.edu/clowder/api/extractors/ncsa.image.ocr"
      }
    },
    {
      "content": {
        "basic_caltech101_score": "-0.696487",
        "basic_caltech101_category": "bonsai"
      },
      "@context": [
        "https://clowder.ncsa.illinois.edu/contexts/metadata.jsonld",
        {
          "basic_caltech101_score": "http://clowder.ncsa.illinois.edu/metadata/ncsa.cv.caltech101#basic_caltech101_score",
          "basic_caltech101_category": "http://clowder.ncsa.illinois.edu/metadata/ncsa.cv.caltech101#basic_caltech101_category"
        }
      ],
      "created_at": "Tue Jun 20 10:33:50 CDT 2017",
      "agent": {
        "extractor_id": "http://141.142.211.76:9000/clowder-dev/extractors/ncsa.cv.caltech101/1.0",
        "@type": "cat:extractor",
        "name": "http://141.142.211.76:9000/clowder-dev/extractors/ncsa.cv.caltech101/1.0"
      }
    },
    {
      "content": {
        "versus_descriptors": [
          {
            "descriptor_id": "class edu.illinois.ncsa.versus.descriptor.impl.RGBHistogramDescriptor",
            "bands": 3,
            "extractor_id": "edu.illinois.ncsa.versus.extract.impl.RGBHistogramExtractor",
            "bins": 8,
            "descriptor": [
              [
                0.5913427300480177,
                0.10906269800126721,
                0.29938380696545547,
                0.00021076498525954203,
                0,
                0,
                0,
                0
              ],
              [
                0.6151575370082368,
                0.18266582360672562,
                0.2015070350995701,
                0.000669604285467427,
                0,
                0,
                0,
                0
              ],
              [
                0.5999196213037582,
                0.1472782516717198,
                0.2524234700556629,
                0.0003786569688591462,
                0,
                0,
                0,
                0
              ]
            ]
          }
        ]
      },
      "@context": [
        "https://clowder.ncsa.illinois.edu/contexts/metadata.jsonld",
        {
          "descriptor_id": "http://clowder.ncsa.illinois.edu/ncsa.versus.image#descriptor_id",
          "versus_descriptors": "http://clowder.ncsa.illinois.edu/ncsa.versus.image#versus_descriptors",
          "bands": "http://clowder.ncsa.illinois.edu/ncsa.versus.image#bands",
          "descriptor": "http://clowder.ncsa.illinois.edu/ncsa.versus.image#descriptor",
          "extractor_id": "http://clowder.ncsa.illinois.edu/ncsa.versus.image#extractor_id",
          "bins": "http://clowder.ncsa.illinois.edu/ncsa.versus.image#bins"
        }
      ],
      "created_at": "Tue Jun 20 10:33:49 CDT 2017",
      "agent": {
        "extractor_id": "https://clowder.ncsa.illinois.edu/clowder/api/extractors/ncsa.versus.image",
        "@type": "cat:extractor",
        "name": "https://clowder.ncsa.illinois.edu/clowder/api/extractors/ncsa.versus.image"
      }
    },
    {
      "content": {
        "Border color": "srgb(223,223,223)",
        "Compression": "Zip",
        "Elapsed time": "0:01.219",
        "Artifacts": {
          "verbose": true,
          "filename": "/tmp/tmpzdHVT2.png"
        },
        "height": 1929,
        "Interlace": "None",
        "Compose": "Over",
        "Tainted": false,
        "Filesize": "3.797MB",
        "Pixels per second": "13.89MB",
        "Units": "PixelsPerCentimeter",
        "Chromaticity": {
          "red primary": "(0.64,0.33)",
          "green primary": "(0.3,0.6)",
          "blue primary": "(0.15,0.06)",
          "white point": "(0.3127,0.329)"
        },
        "Type": "TrueColor",
        "Rendering intent": "Perceptual",
        "Print size": "22.3539x27.2227",
        "Channel depth": {
          "blue": "8-bit",
          "green": "8-bit",
          "red": "8-bit"
        },
        "Background color": "white",
        "Transparent color": "black",
        "Format": "PNG (Portable Network Graphics)",
        "Image statistics": {
          "Overall": {
            "skewness": 0.67031,
            "min": "0 (0)",
            "max": "255 (1)",
            "standard deviation": "73.7679 (0.289286)",
            "kurtosis": -1.19763,
            "mean": "91.9025 (0.360402)"
          }
        },
        "Number pixels": "3.056M",
        "width": 1584,
        "Channel statistics": {
          "Blue": {
            "skewness": 0.771814,
            "min": "0 (0)",
            "max": "255 (1)",
            "standard deviation": "70.8725 (0.277931)",
            "kurtosis": -0.940633,
            "mean": "87.3909 (0.342709)"
          },
          "Green": {
            "skewness": 0.627148,
            "min": "0 (0)",
            "max": "255 (1)",
            "standard deviation": "73.9244 (0.2899)",
            "kurtosis": -1.27569,
            "mean": "90.7096 (0.355724)"
          },
          "Pixels": 3055536,
          "Red": {
            "skewness": 0.59793,
            "min": "0 (0)",
            "max": "255 (1)",
            "standard deviation": "76.4027 (0.299618)",
            "kurtosis": -1.38753,
            "mean": "97.607 (0.382772)"
          }
        },
        "Class": "DirectClass",
        "Mime type": "image/png",
        "Page geometry": "1584x1929+0+0",
        "Geometry": "1584x1929+0+0",
        "Colorspace": "sRGB",
        "Depth": "8-bit",
        "User time": "0.130u",
        "Matte color": "grey74",
        "Resolution": "70.86x70.86",
        "Gamma": 0.45455,
        "Image": "/tmp/tmpzdHVT2.png",
        "Properties": {
          "photoshop": {
            "DateCreated": "2014-01-13T09:13:13"
          },
          "exif": {
            "LightSource": 0,
            "FlashPixVersion": "48, 49, 48, 48",
            "ExposureMode": 0,
            "Flash": 16,
            "SceneCaptureType": 0,
            "DateTime": "2014:01:13 09:13:13",
            "MeteringMode": 5,
            "ExifVersion": "48, 50, 51, 48",
            "ExposureBiasValue": "0/1",
            "Saturation": 0,
            "ExposureProgram": 2,
            "FocalLengthIn35mmFilm": 33,
            "ColorSpace": 1,
            "ExifImageWidth": 1584,
            "Contrast": 0,
            "DateTimeOriginal": "2014:01:13 09:13:13",
            "WhiteBalance": 0,
            "CompressedBitsPerPixel": "4/1",
            "DateTimeDigitized": "2014:01:13 09:13:13",
            "SensingMethod": 2,
            "FNumber": "7/2",
            "CustomRendered": 0,
            "FocalLength": "28/5",
            "ComponentsConfiguration": "1, 2, 3, 0",
            "ExifOffset": 188,
            "ISOSpeedRatings": 100,
            "Model": "DMC-FZ200",
            "Software": "Ver.1.0",
            "ExposureTime": "1/125",
            "FileSource": 3,
            "MaxApertureValue": "95/32",
            "Make": "Panasonic",
            "Sharpness": 0,
            "GainControl": 0,
            "DigitalZoomRatio": "0/1",
            "ExifImageLength": 1929
          },
          "icc": {
            "model": "IEC 61966-2.1 Default RGB colour space - sRGB",
            "description": "sRGB IEC61966-2.1",
            "copyright": "Copyright (c) 1998 Hewlett-Packard Company",
            "manufacturer": "IEC http://www.iec.ch"
          },
          "signature": "2decf705da4c929f51db201f646f51f5dcd9dcfd42aac6997cf825e37593ca8d",
          "date": {
            "create": "2017-06-20T15:33:48+00:00",
            "modify": "2017-06-20T15:33:48+00:00"
          },
          "aux": {
            "SerialNumber": "O071309260157"
          },
          "xmp": {
            "CreateDate": "2014-01-13T09:13:13",
            "ModifyDate": "2014-01-13T09:13:13",
            "CreatorTool": "Ver.1.0"
          },
          "png": {
            "bKGD": "chunk was found (see Background color, above)",
            "IHDR": {
              "color-type-orig": 2,
              "color_type": "2 (Truecolor)",
              "bit_depth": 8,
              "width,height": "1584, 1929",
              "bit-depth-orig": 8,
              "interlace_method": "0 (Not interlaced)"
            },
            "pHYs": "x_res=7086, y_res=7086, units=1",
            "text": "47 tEXt/zTXt/iTXt chunks were found",
            "sRGB": "intent=0 (Perceptual Intent)",
            "gAMA": "gamma=0.45455 (See Gamma, above)",
            "tIME": "2017-05-08T17:53:53Z",
            "cHRM": "chunk was found (see Chromaticity, above)"
          }
        },
        "Version": "ImageMagick 6.8.9-9 Q16 x86_64 2017-05-26 http://www.imagemagick.org",
        "Iterations": 0
      },
      "@context": {
        "@vocab": "http://www.w3.org/2003/12/exif/ns"
      },
      "created_at": "Tue Jun 20 10:33:48 CDT 2017",
      "agent": {
        "extractor_id": "http://141.142.211.76:9000/clowder-dev/api/extractors/ncsa.image.metadata",
        "@type": "cat:extractor",
        "name": "http://141.142.211.76:9000/clowder-dev/api/extractors/ncsa.image.metadata"
      }
    },
    {
      "content": {
        "conformsTo": [
          {
            "sf:mime": "image/png",
            "sf:version": "1.1",
            "@id": "info:pronom/fmt/12",
            "sf:name": "Portable Network Graphics",
            "sf:basis": "extension match png; byte match at [[[0 16]] [[37 4]] [[3796773 12]]] (signature 3/3)"
          }
        ],
        "extent": 3796785
      },
      "@context": [
        "https://clowder.ncsa.illinois.edu/contexts/metadata.jsonld",
        {
          "conformsTo": "http://purl.org/dc/terms/conformsTo",
          "extent": "http://purl.org/dc/terms/extent"
        }
      ],
      "created_at": "Tue Jun 20 10:33:48 CDT 2017",
      "agent": {
        "extractor_id": "http://141.142.211.76:9000/clowder-dev/extractors/siegfried/2.0",
        "@type": "cat:extractor",
        "name": "http://141.142.211.76:9000/clowder-dev/extractors/siegfried/2.0"
      }
    },
    {
      "content": {
        "mean grayscale value": 92.37236772860801
      },
      "@context": [
        "https://clowder.ncsa.illinois.edu/contexts/metadata.jsonld",
        {
          "mean grayscale value": "http://clowder.ncsa.illinois.edu/ncsa.cv.meangrey#mean_grayscale_value"
        }
      ],
      "created_at": "Tue Jun 20 10:33:48 CDT 2017",
      "agent": {
        "extractor_id": "https://clowder.ncsa.illinois.edu/clowder/api/extractors/ncsa.cv.meangrey",
        "@type": "cat:extractor",
        "name": "https://clowder.ncsa.illinois.edu/clowder/api/extractors/ncsa.cv.meangrey"
      }
    },
    {
      "content": {
        "sha1": "cd89ff5f85078b3a98f4f92dc25894a69bd89181",
        "sha224": "9b9f30bede741663dcb7c3d1f1d9427a3b059d490831c3a6cd6b62ca",
        "sha384": "5ec428a0f040935aca0ff21fa72478864626ecd8ce326db12a7ede223464126a1a91ff5712192af8df9051ea7ab3ee4c",
        "sha256": "12adc20571b87d90350238f16a5fcaeaa9d5ea56cd3a4e57616d84b8d2799a32",
        "sha512": "c7dd3508b864904de193b105d4d2347ec591fa31d0b6ab54bcd3fa2c2ecef9967bb14a452d434d21fb5cfacc849d74c9e098648671e1ec1776599ebb0931b28e",
        "md5": "2a2d9dc555876abfd4641c4529eb7960"
      },
      "@context": [
        "https://clowder.ncsa.illinois.edu/contexts/metadata.jsonld",
        {
          "sha1": "http://www.w3.org/2001/04/xmldsig-more#sha1",
          "sha224": "http://www.w3.org/2001/04/xmldsig-more#sha224",
          "sha384": "http://www.w3.org/2001/04/xmldsig-more#sha384",
          "sha256": "http://www.w3.org/2001/04/xmldsig-more#sha256",
          "sha512": "http://www.w3.org/2001/04/xmldsig-more#sha512",
          "md5": "http://www.w3.org/2001/04/xmldsig-more#md5"
        }
      ],
      "created_at": "Tue Jun 20 10:33:48 CDT 2017",
      "agent": {
        "extractor_id": "http://141.142.211.76:9000/clowder-dev//api/extractors/ncsa.file.digest",
        "@type": "cat:extractor",
        "name": "http://141.142.211.76:9000/clowder-dev//api/extractors/ncsa.file.digest"
      }
    }
  ],
  "versusmetadata": [
    {
      "descriptor_id": "class edu.illinois.ncsa.versus.descriptor.impl.RGBHistogramDescriptor",
      "bands": 3,
      "extractor_id": "edu.illinois.ncsa.versus.extract.impl.RGBHistogramExtractor",
      "bins": 8,
      "descriptor": [
        [
          0.5913427300480177,
          0.10906269800126721,
          0.29938380696545547,
          0.00021076498525954203,
          0,
          0,
          0,
          0
        ],
        [
          0.6151575370082368,
          0.18266582360672562,
          0.2015070350995701,
          0.000669604285467427,
          0,
          0,
          0,
          0
        ],
        [
          0.5999196213037582,
          0.1472782516717198,
          0.2524234700556629,
          0.0003786569688591462,
          0,
          0,
          0,
          0
        ]
      ]
    }
  ],
  "tags": [],
  "filename": "ncsa.png",
  "technicalmetadata": [
    {
      "Human Preference Extractor": {
        "Definitions": {
          "Human Preference": "A Computer Vision model that uses the spectral data of an image to get a human preference value ranging from 1 to 5.",
          "Green Index": "The green index is the estimated percentage of green pixels within an image."
        },
        "Data": {
          "Green Index": "2.3"
        }
      }
    }
  ],
  "id": "5949405b4f0c4db82da2da84"
}

Optionally you can use the -v flag to get a more verbose output, including the content of the output file ncsa.json.

You can also get a list of available extractors that will process a given input file:

Conversion

In Brown Dog, conversion refers to the processing of converting files from one format to another. For example converting an image from PNG to JPEG format. You can find an example of the extraction command below:

TODO: Attach screenshot for extraction

Chaining Transformations


Help

Below you can see the output of the command (bd -h) which basically describes how to use this client:

BD-CLI Help
Brown Dog Command-Line Interface (BD-CLI)

=========================================

bd [OPTION] [FILE/DIRECTORY]

-----------------------------

Command line interface to Brown Dog services. Use to convert files between formats and extract various derived products

from file contents (e.g. tags, metadata, signatures, previews).

-b set the BD URL (e.g. https://bd-api.ncsa.illinois.edu)

-t set the token

-v verbose output

-w set wait time for conversions/extractions

-h display help

-o set output format

--outputs list the available output formats for the given file

--extractors list the available extractors for the given file

--find search current directory for a file similar to the given file

--bigdata use this flag if the file is large enough and it needs to be processed locally rather than using remote

 Brown Dog services

 **Examples:**

List available output formats for given file:

  bd --outputs image.pcd

Convert the given Kodak Photo CD image to a JPEG image.  Produces file image.jpg:

  bd -o jpg image.pcd

Convert a directory of images into the JPEG format:

  bd -o jpg images/

List available extractors for given file:

  bd --extractors image.jpg

Extract data from files contents (e.g. tags, previews, analysis, and other derived products).  Produces file image.json:

  bd image.jpg

Chain conversion and extraction.  Equivalent to the previous two examples:

  bd -o jpg image.pcd | bd

Index files within a directory:

  bd images/

Search for a similar file within a directory that has already been indexed:

  bd --find /path/image.jpg



  • No labels