Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Anchor

...

StartHere

...

StartHere
Start Here

To begin does your code, software, or tool carry out a data conversion or a data extraction?  If a conversion the tool should be included in the Data Access Proxy.  If an extraction the tool should be included in the Data Tilling Service.

Anchor
DAP
DAP
The Data Access Proxy (DAP)

The Data Access Proxy handles data conversions.  If a piece of software or tool exists to carry out the conversion its incorporation into the DAP will be through Polyglot.  If the specification of the file format is known then in can be incorporated as a DFDL schema within Daffodil.

Anchor
Polyglot
Polyglot
Polyglot Software Server Scripts

Software Server scripts are used by Polyglot to automate the interaction with software that is capable of converting from one file format to another.  These scripts can directly wrap command line utilities that carry out conversions for use in Polyglot or split the steps of opening a file in one format and saving a file in a different format, typical of GUI driven applications.  These wrapper scripts can be written in pretty much any text based scripting language.  Below we show a few simple examples.  Full details on the creation of these wrapper scripts, the required naming conventions, and required header conventions please refer to the the Scripting Manual.

Anchor
CommandLIne
CommandLIne
Command Line Applications

Bash Script

The following is an example of a bash wrapper script for ImageMagick.  Note that it is fairly straight forward.  The comments at the top contain the information Polyglot needs to use the application: the name and version of the application, they type of data it supports, the input formats it supports, and the output formats it supports.

Code Block
titleImgMgk_convert.sh
#!/bin/sh
#ImageMagick (v6.5.2)
#image
#bmp, dib, eps, fig, gif, ico, jpg, jpeg, pdf, pgm, pict, pix, 

Software Server scripts are used by Polyglot to automate the interaction with software that is capable of converting from one file format to another.  These scripts can directly wrap command line utilities that carry out conversions for use in Polyglot or split the steps of opening a file in one format and saving a file in a different format, typical of GUI driven applications.  These wrapper scripts can be written in pretty much any text based scripting language.  Below we show a few simple examples.  Full details on the creation of these wrapper scripts, the required naming convensions, and required header convensions please refer to the the Scripting Manual.

AnchorCommandLIneCommandLIneCommand Line Applications

Bash Script

The following is an example of a bash wrapper script for ImageMagick.  Note that it is fairly straight forward.  The comments at the top contain the information Polyglot needs to use the application: the name and version of the application, they type of data it supports, the input formats it supports, and the output formats it supports.

Code Block
titleImgMgk_convert.sh
#!/bin/sh
#ImageMagick (v6.5.2)
#image
#bmp, dib, eps, fig, gif, ico, jpg, jpeg, pdf, pgm, pict, pix, png, pnm, ppm, ps, rgb, rgba, sgi, sun, svg, tga, tif, tiff, ttf, x, xbm, xcf, xpm, xwd, yuv
#bmp, dib, eps, gif, jpg, jpeg, pdf, pgm, pict, png, pnm, ppm, ps, rgb, rgba, sgi, sun, svg, tga, tif, tiff, ttf, x, xbm, xpm, xwd, yuv

convert $1 $2

Batch File

Some GUI based applications are capable of being called in a headless mode.  The following is an example wrapper script for OpenOffice called in its headless mode.

Code Block
titleOpenOffice_convert.bat
REM OpenOffice (v3.1.0)
REM document
REM doc, odt, rtf, txt
REM doc, odt, pdf, rtf, txt

"C:\Program Files\OpenOffice.org 3\program\soffice.exe" -headless -norestore "-accept=socket`,host=localhost`,port=8100;urp;StarOffice.ServiceManager"
"C:\Program Files\OpenOffice.org 3\program\python.exe" "C:\Converters\DocumentConverter.py" "%1%" "%2%"

Anchor
GUI
GUI
GUI Applications

AutoHotKey

The following is an example of an AutoHotKey script to convert files with Adobe Acrobat, a GUI driven application.  Note it contains a similar header in the comments at the beginning of the script.  Also note that the open and save operation can be broken into two separate scripts.

...

Code Block
titlepgm.dfdl.xsd
<?xml version="1.0" encoding="UTF-8"?>

<!--
Load image data from a PGM file and represent the data as a sequence of pixels in row major order.
-->

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/" xmlns:ex="http://example.com" targetNamespace="http://example.com">
  <xs:include schemaLocation="xsd/built-in-formats.xsd"/>

  <xs:annotation>
    <xs:appinfo source="http://www.ogf.org/dfdl/">
      <dfdl:format ref="ex:daffodilTest1" separator="" initiator="" terminator="" leadingSkip='0' textTrimKind="none" initiatedContent="no"
        alignment="implicit" alignmentUnits="bits" trailingSkip="0" ignoreCase="no" separatorPolicy="suppressed" 
        separatorPosition="infix" occursCountKind="parsed" emptyValueDelimiterPolicy="both" representation="text" 
        textNumberRep="standard" lengthKind="delimited" encoding="ASCII"/>
    </xs:appinfo>
  </xs:annotation>

  <xs:element name="file">
    <xs:complexType>
      <xs:sequence>

        <xs:element name="header" dfdl:lengthKind="implicit" maxOccurs="1">
          <xs:complexType>
            <xs:sequence dfdl:sequenceKind="ordered" dfdl:separator="%NL;" dfdl:separatorPosition="postfix">
              <xs:element name="type" type="xs:string"/>
              <xs:element name="dimensions" maxOccurs="1" dfdl:occursCountKind="implicit">
                <xs:complexType>
                  <xs:sequence dfdl:sequenceKind="ordered" dfdl:separator="%SP;">
                    <xs:element name="width" type="xs:integer"/>
                    <xs:element name="height" type="xs:integer"/>
                  </xs:sequence>
                </xs:complexType>
              </xs:element>
              <xs:element name="depth" type="xs:integer"/>
            </xs:sequence>
          </xs:complexType>
        </xs:element>

        <xs:element name="pixels" dfdl:lengthKind="implicit" maxOccurs="1">
          <xs:complexType>
            <xs:sequence dfdl:separator="%SP; %NL; %SP;%NL;" dfdl:separatorPosition="postfix" dfdl:separatorSuppressionPolicy="anyEmpty">
              <xs:element name="pixel" type="xs:integer" maxOccurs="unbounded"/>" dfdl:occursCountKind="expression"
            </xs:sequence>
          </xs:complexType>
        </xs:dfdl:occursCount="{../../ex:header/ex:dimensions/ex:width * ../../ex:header/ex:dimensions/ex:height }"/>
            </xs:sequence>
          </xs:complexType>
        </xs:element>

      </xs:sequence>
    </xs:complexType>
  </xs:element>

</xs:schema>

...

Code Block
<ex:file xmlns:ex="http://example.com">
  <ex:header>
    <ex:type>P2</ex:type>
    <ex:dimensions>
      <ex:width>16</ex:width>
      <ex:height>16</ex:height>
    </ex:dimensions>
    <ex:depth>255</ex:depth>
  </ex:header>
  <ex:pixels>
    <ex:pixel>136</ex:pixel>
    <ex:pixel>136</ex:pixel>
    <ex:pixel>136</ex:pixel>
    ...
    <ex:pixel>136</ex:pixel>
    <ex:pixel>136</ex:pixel>
  </ex:pixels>
</ex:file>

Anchor

...

DTS

...

DTS
The Data Tilling Service (DTS)

The Data Tilling Services handles data extractions.  If your code, tool, or software extracts information such as keywords from a file or its contents then it should be included in the DTS as a Medici extractor.  If your code, tool, or software extracts a signature from the file's contents which in turn can be compared to the signatures of other files via some distance measure to find similar pieces of data, then, it should be included in the DTS as a Versus extractor.

Anchor
Medici
Medici
Medici Extractors

Medici extractors typically serve to automatically extract some new kind of information from a file's content when it is uploaded into Medici.  These extractors do this by connecting to a shared RabbitMQ bus.  When a new file is uploaded to Medici it is Medici extractors typically serve to automatically extract some new kind of information from a file's content when it is uploaded into Medici.  These extractors do this by connecting to a shared RabbitMQ bus.  When a new file is uploaded to Medici it is announced on this bus.  Extractors that can handle a file of the type posted on the bus are triggered and the data they in turn create is returned to Medici as derived data to be associated with that file.  The extractors themselves can be implemented in a variety of languages. Examples of these extractors in different languages can be found in the extractors-templates code repository.

Anchor
Anchor
Java
Java
Java

Code Block
themeEmacs
languagejava
titleConnecting to RabbitMQ
protected void startExtractor(String rabbitMQUsername,
	String rabbitMQpassword) {
	try{ 
 		//Open channel and declare exchange and consumer
		ConnectionFactory factory = new ConnectionFactory();
		factory.setHost(serverAddr);
		factory.setUsername(rabbitMQUsername);
		factory.setPassword(rabbitMQpassword);
		Connection connection = factory.newConnection();

 		final Channel channel = connection.createChannel();
		channel.exchangeDeclare(EXCHANGE_NAME, "topic", true);

		channel.queueDeclare(QUEUE_NAME,DURABLE,EXCLUSIVE,AUTO_DELETE,null);
		channel.queueBind(QUEUE_NAME, EXCHANGE_NAME, "*.file.text.plain.#");
 
 		this.channel = channel;

 		// create listener
		channel.basicConsume(QUEUE_NAME, false, CONSUMER_TAG, new DefaultConsumer(channel) {
 			@Override
 			public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
				messageReceived = new String(body);
 				long deliveryTag = envelope.getDeliveryTag();
 				// (process the message components here ...)
				System.out.println(" {x} Received '" + messageReceived + "'");
 
				replyProps = new AMQP.BasicProperties.Builder().correlationId(properties.getCorrelationId()).build();
				replyTo = properties.getReplyTo();
 
				processMessageReceived();
				System.out.println(" [x] Done");
				channel.basicAck(deliveryTag, false);
			}
		});

 		// start listening 
		System.out.println(" [*] Waiting for messages. To exit press CTRL+C");
 		while (true) {
			Thread.sleep(1000);
		}
	}
 	catch(Exception e){
		e.printStackTrace();
		System.exit(1);
	} 
}

Java extractors can be created using the amqp-client jar file. This allows you to connect to the RabitMQ bus and received messages. The easiest way to get up and running is to use maven with java to add all required dependencies. An example of an extractor written in java can be found at Medici Extractor in Java.

Anchor
Python
Python
Python

Python extractors will often be based on the packages pika and requests. This allows you to connect to the RabittMQ message bus and easily send requests to medici. A complete example of the python extractor can be found at Medici Extractor in Python

Calling R Scripts from Python

Coming soon...

Anchor
Versus
Versus
Versus Extractors

Versus extractors serve to extract a signature from a file's content.  These signatures, effectively a hash for the data, are typically numerical vectors which capture some semantically meaningful aspect of the content so that two such signatures can then be compared using some distance measure.  Within Versus extractors operate on a data structure representing the content of a file, produced a Versus adapter, and the returned signatures compared by either a Versus similarity or distance measure.  The combination of these adapters, extractors, and measures in turn compose a comparison which can be used for relating files according their contents.

Anchor
Java Measure
Java Measure
Java

The main class sets up the comparison, this is done by adding the two files that need to be compared, as well as the adapter to load the file, the extractor to extract a feature from the file, and a measurement to compare the two features.

Code Block
languagejava
titleMain
    static public void main(String[] args) {         
        PairwiseComparison comparison = new PairwiseComparison();
        comparison.setId(UUID.randomUUID().toString());
        comparison.setFirstDataset(new File("data/test1.txt"));
        comparison.setSecondDataset(new File("data/test2.txt"));
        comparison.setAdapterId(TextAdapter.class.getName());
        comparison.setExtractorId(TextHistogramExtractor.class.getName());
        comparison.setMeasureId(LabelHistogramEuclidianDistanceMeasure.class.getName());

        ExecutionEngine ee = new ExecutionEngine(); 
        ee.submit(comparison, new ComparisonStatusHandler() {
            @Override
            public void onStarted() {
                System.out.println("STARTED : "
Code Block
languagejava
titleProcessing Messages Received From RabbitMQ
protected void processMessageReceived() {
  try {
    try {
      ExampleJavaExtractorService extrServ = new ExampleJavaExtractorService(this);
      jobReceived = getRepresentation(messageReceived, ExtractionJob.class);   }

     
    
   @Override
           File textFilepublic =void extrServ.processJob(jobReceived);
onFailed(String msg, Throwable e) {
          
      jobReceivedSystem.out.setFlagprintln("wasText"FAILED  : " + msg);

      log.info("Word count extraction complete. Returning word count file as intermediate resulte."printStackTrace();
             sendStatus(jobReceived.getId(), this.getClass().getSimpleName(), "Word count extraction complete. Returning word count file as intermediate result.", log);   System.exit(0);
            }

            @Override
        
    public void uploadIntermediateonDone(textFile, "text/plain", log);double value) {
        
        textFileSystem.out.delete(println("DONE    : " + value);
        
        sendStatus(jobReceivedSystem.getId(), this.getClass().getSimpleName(), "DONE.", log);
exit(0);
    } catch (Exception ioe) {
    }

  log.error("Could not finish extraction job.", ioe);
      sendStatus(jobReceived.getId(), this.getClass().getSimpleName(), "Could not finish extraction job.", log);
@Override
            public void sendStatus(jobReceived.getId(), this.getClass().getSimpleName(), "DONE.", log);
onAborted(String msg) {
     }
  } catch(Exception e) {
      eSystem.out.printStackTrace(println("ABORTED : " + msg);
                System.exit(10);
  } 
}
Code Block
languagejava
titleProcessing the Job
public File processJob(ExtractionJob receivedMsg) throws Exception{
      }
  log.info("Downloading text file with ID "+ receivedMsg.getIntermediateId() +" from " + receivedMsg.getHost());
  callingExtractor.sendStatus(receivedMsg.getId(), callingExtractor.getClass().getSimpleName(), "Downloading text file.", log);
      
  DefaultHttpClient httpclient = new DefaultHttpClient();
  HttpGet httpGet = new HttpGet(receivedMsg.getHost() +"api/files/"+ receivedMsg.getIntermediateId()+"?key="+playserverKey);    
  HttpResponse fileResponse = httpclient.execute(httpGet);
  log.info(fileResponse.getStatusLine());
  if(fileResponse.getStatusLine().toString().indexOf("200") == -1){
    throw new IOException("File not found.");
  }
  HttpEntity fileEntity = fileResponse.getEntity();
  InputStream fileIs = fileEntity.getContent();
  
  Header[] hdrs = fileResponse.getHeaders("content-disposition");
  String contentDisp = hdrs[0].toString();
  
  String fileName = contentDisp.substring(contentDisp.indexOf("filename=")+9);
  File tempFile = File.createTempFile(fileName.substring(0, fileName.lastIndexOf(".")),      fileName.substring(fileName.lastIndexOf(".")).toLowerCase());
  OutputStream fileOs = new FileOutputStream(tempFile);   
  IOUtils.copy(fileIs,fileOs);
  fileIs.close();
  fileOs.close });
    }

The text adapter will take a text file, and load all the file, splitting the text into words and return a list of all words in the text. The words are still in the right order, and it is possible to read the original information of the file by reading the words in the order as they are returned by getWords().

Code Block
languagejava
titleText Adapter
public class TextAdapter implements FileLoader, HasText {
    private File         file;
    private List<String> words;

    public TextAdapter() {}

    // ----------------------------------------------------------------------
    // FileLoader
    // ----------------------------------------------------------------------
    @Override
    public void load(File file) {
        this.file = file;
    }

    @Override
    public String getName() {
        return "Text Document";
    }

    @Override
    public List<String> getSupportedMediaTypes() {
        List<String> mediaTypes = new ArrayList<String>();
      
  EntityUtilsmediaTypes.consume(fileEntityadd("text/*");
  
      return mediaTypes;
  
  log.info("Download complete. Initiating word count generation");
  
  File textFile = processFile(tempFile, receivedMsg.getId());  
  return textFile;       
}
Code Block
languagejava
titleProcessing the file
private File processFile(File tempFile, String originalFileId) throws Exception  {
    
    Runtime r = Runtime.getRuntime}

    // ----------------------------------------------------------------------
    // HasText
    // ----------------------------------------------------------------------
    @Override
    public List<String> getWords() {
        if (words == null) {
            words = new ArrayList<String>();
    Process    p;    try //{
 Process tracks one external native process
      
    StringBufferedReader tempDirbr = System.getProperty("java.io.tmpdir"new BufferedReader(new FileReader(file));
    if (new Character(tempDir.charAt(tempDir.length()-1)).toString().equals(System.getProperty("file.separator")) == false){
      tempDir = tempDir + System.getProperty("file.separator")String line;
    }
       
    String processCmdwhile((line = "";
    String operSystem = System.getProperty("os.name").toLowerCase();

	// TODO: windows impl
    if(operSystem.indexOf("nix") >= 0 || operSystem.indexOf("nux") >= 0 || operSystem.indexOf("aix") > 0 ){
br.readLine()) != null) {
                    String[] w = line.split(" ");
             "wc -w " + tempDir +  tempFile.getName();
words.addAll(Arrays.asList(w));
      }    
      }
    p = r.exec(processCmd, null, new File(tempDir));
    StreamGobbler outputGobbler = new StreamGobbler(pbr.getInputStream(), "INFO", logclose();
    StreamGobbler  errorGobbler = new StreamGobbler(p.getErrorStream(),"ERROR", log);
  } catch outputGobbler.start(IOException e); {
    errorGobbler.start();
    p.waitFor();
        e.printStackTrace();
    File outFile = new File(tempDir + tempFile.getName().substring(0, tempFile.getName().lastIndexOf(".")) + ".txt"); }
    tempFile.delete();
    if(!Files.exists(outFile.toPath()))}
      throw new Exception("File not processed correctly. File is possibly corrupt.");
      
  return outFile;
      
}

...

return words;
    }
 }

The extractor will take the words returned by the adapter and count the occurrence of each word. At this point we are left with a histogram with all words and how often they occur in the text, we can no longer read the text since the information about the order of the words is lost.

Code Block
languagejava
titleText Histogram Extractor
public class TextHistogramExtractor implements Extractor
{
    @Override
    public Adapter newAdapter() {
        throw (new RuntimeException("Not supported."));
    }

    @Override
    public String getName() {
        return "Text Histogram Extractor";
    }

    @Override
    public Set<Class<? extends Adapter>> supportedAdapters() {
        Set<Class<? extends Adapter>> adapters = new HashSet<Class<? extends Adapter>>();

...

Code Block
languagecpp
titleConnecting to RabbitMQ
#include <amqpcpp.h>

namespace CPPExample {

  class RabbitMQConnectionHandler : public AMQP::ConnectionHandler {
      /**
      *  Method that is called by the AMQP library every time it has data
      *  available that should be sent to RabbitMQ. 
      *  @param  connection  pointer to the main connection object  
      *  @paramadapters.add(HasText.class);
  data      return adapters;
 memory buffer with the}

 data that should be@Override
 sent to RabbitMQ
 public Class<? extends Descriptor> getFeatureType() *{
  @param  size    return LabelHistogramDescriptor.class;
   size of}

 the buffer
  @Override
    */
public Descriptor extract(Adapter adapter) throws virtualException void onData(AMQP::Connection *connection, const char *data, size_t size)
    {
        if (adapter instanceof HasText) {
         // @todo 
 LabelHistogramDescriptor desc =    new LabelHistogramDescriptor();

  //  Add your own implementation, for example by doing for a(String callword to the
  : ((HasText) adapter).getWords()) {
       //  send() system call. But be aware that the senddesc.increaseBin() call may notword);
         //  send all}

 data at once, so you also need to take care of return bufferingdesc;
         //} else {
  the bytes that could not immediately be sent, and try tothrow sendnew UnsupportedTypeException();
        }
 //  them again}
 when the socket becomes writable again
     }@Override

    public  /**
boolean hasPreview(){
       * return Methodfalse;
 that is called by}
 the AMQP library when
 the login attempt @Override
    public String *previewName(){
  succeeded. After this method has been called, the connection is ready 
      *  to use.return null;
    }
}

To compare two texts we use the euclidian distance measure of two histograms. First we normalize each histogram, so we can compare a large text with a small text, next we compare each big of the two histograms. If the bin is missing from either histogram it is assumed to have a value of 0.

Code Block
languagejava
titleEuclidian Distance Measure
public class LabelHistogramEuclidianDistanceMeasure implements Measure 
{
    @Override
    public SimilarityPercentage *normalize(Similarity similarity) @param{
  connection      Thereturn connectionnew thatSimilarityPercentage(1 can now be used
  - similarity.getValue());
    }

    */@Override
    public String virtual void onConnected(Connection *connection)getFeatureType() {
      {
  return LabelHistogramDescriptor.class.getName();
    }

   // @todo@Override
    public String getName() {
  //  add your own implementation, forreturn example by creating a channel "Histogram Distance";
    }

    @Override
     //public Class<LabelHistogramEuclidianDistanceMeasure> getType() {
  instance, and start publishing or consuming
 return LabelHistogramEuclidianDistanceMeasure.class;
    }

      /**// correlation

    @Override
  *  Methodpublic thatSimilarity is called by the AMQP library when a fatal error occurs
compare(Descriptor desc1, Descriptor desc2) throws Exception {
        if ((desc1 *instanceof LabelHistogramDescriptor) on&& the(desc2 connection, for example because data received from RabbitMQ
instanceof LabelHistogramDescriptor)) {
           * LabelHistogramDescriptor couldlhd1 not= be recognized.(LabelHistogramDescriptor) desc1;
      *  @param  connection  LabelHistogramDescriptor lhd2 =  The connection on which the error occured
(LabelHistogramDescriptor) desc2;

            *// get @paramall possible messagelabels
         A human readable errorSet<String> message
labels = new HashSet<String>();
   */
      virtual void onError(Connection *connection, const std::string &message) labels.addAll(lhd1.getLabels());
      {
      labels.addAll(lhd2.getLabels());

   // @todo
        // normalize
 add your own implementation, for example by reporting the error
  lhd1.normalize();
      //  to the user of your program, log the error, and destruct the 
 lhd2.normalize();
            
    //  connection object because it is no longer in a usable state
// compute distance
         }
  };

}
Code Block
languagecpp
titleReceiver
namespace CPPExample {

  /** double sum = 0;

   *  Parse data that was recevied from RabbitMQ
 for (String s *: labels) {
   *  Every time that data comes in from RabbitMQ, you should call thisDouble methodb1 to parse= lhd1.getBin(s);
   *  the incoming data, and let it handle by the AMQP-CPP library. ThisDouble methodb2 returns the number
= lhd2.getBin(s);
    *  of bytes that were processed.
   *
   *  If not all bytes could be processed because it only contained a if partial frame, you should(b1 == null) {
   *  call this same method later on when more data is available. The AMQP-CPP library does notsum do
+= b2 * b2;
 *  any buffering, so it is up to the caller to ensure that the old} dataelse isif also(b2 passed== innull) that{
   *  later call.
   *
   *  @param  buffer    sum += bufferb1 to* decodeb1;
   *  @param  size        size of} the buffer to decodeelse {
   *  @return             number of bytessum that+= were(b1 processed
- b2)  */
 (b1 size_t parse(char *buffer, size_t size)
  {- b2);
     return _implementation.parse(buffer, size);
  }
}

...

Code Block
themeEmacs
languagepy
titleInstantiating the logger and starting the extractor
def main():
 global logger

  # name of receiver}
  receiver='ExamplePythonExtractor'

  # configure the logging system
  logging.basicConfig(format="%(asctime)-15s %(name)-10s %(levelname)-7s : %(message)s", level=logging.WARN)
  logger = logging.getLogger(receiver)
  logger.setLevel(logging.DEBUG)
 
  if len(sys.argv) != 4:
    logger.info("Input RabbitMQ username, followed by RabbitMQ password and Medici REST API key.")
    sys.exit()
 
  global playserverKey
  playserverKey = sys.argv[3]
  global exchange_name
  exchange_name = sys.argv[4]
Code Block
themeEmacs
languagepy
titleConnecting to RabbitMQ
# connect to rabbitmq using input username and password 
credentials = pika.PlainCredentials(sys.argv[1], sys.argv[2])
parameters = pika.ConnectionParameters(credentials=credentials)
connection = pika.BlockingConnection(parameters)
 
 # connect to channel
channel = connection.channel()

 # declare the exchange
channel.exchange_declare(exchange='medici', exchange_type='topic', durable=True)

 # declare the queue
channel.queue_declare(queue=receiver, durable=True)

 # connect queue and exchange
channel.queue_bind(queue=receiver, exchange='medici', routing_key='*.file.text.plain')

 # create listener
channel.basic_consume(on_message, queue=receiver, no_ack=False)

 # start listening
logger.info("Waiting for messages. To exit press CTRL+C")
 try:
   channel.start_consuming()
 except KeyboardInterrupt:
   channel.stop_consuming()

 
# close connection
connection.close()
Code Block
languagepy
titleOn Message
def on_message(channel, method, header, body):
	global logger
	statusreport = {}
	inputfile=None
	try:
		# parse body back from json
		jbody=json.loads(body)

		host=jbody['host']
		fileid=jbody['id']
		intermediatefileid=jbody['intermediateId']
		if not (host.endswith('/')):
			host += '/'
		
		# for status reports
		statusreport['file_id'] = fileid
		statusreport['extractor_id'] = 'wordCount' 

		# print what we are doing
		logger.debug("[%s] started processing", fileid)

		# fetch data
		statusreport['status'] = 'Downloading file.'
		statusreport['start'] = time.strftime('%Y-%m-%dT%H:%M:%S')
		channel.basic_publish(exchange='',
							routing_key=header.reply_to,
							properties=pika.BasicProperties(correlation_id = \
														header.correlation_id),
							body=json.dumps(statusreport)) 
		url=host + 'api/files/' + intermediatefileid + '?key=' + playserverKey
		r=requests.get(url, stream=True)
		r.raise_for_status()
		(fd, inputfile)=tempfile.mkstemp()
		with os.fdopen(fd, "w") as f:
			for chunk in r.iter_content(chunk_size=10*1024):
				f.write(chunk)


		# create word count
		statusreport['status'] = 'Creating word count.'
		statusreport['start'] = time.strftime('%Y-%m-%dT%H:%M:%S')
		channel.basic_publish(exchange='',
							routing_key=header.reply_to,
							properties=pika.BasicProperties(correlation_id = \
														header.correlation_id),
							body=json.dumps(statusreport))
		create_word_count(inputfile, ext, host, fileid)

		# Ack
		channel.basic_ack(method.delivery_tag)
		logger.debug("[%s] finished processing", fileid)
	except subprocess.CalledProcessError as e:
		logger.exception("[%s] error processing [exit code=%d]\n%s", fileid, e.returncode, e.output)
		statusreport['status'] = 'Error processing.'
		statusreport['start'] = time.strftime('%Y-%m-%dT%H:%M:%S') 
		channel.basic_publish(exchange='',
                routing_key=header.reply_to,
                properties=pika.BasicProperties(correlation_id = \
                                                header.correlation_id),
                body=json.dumps(statusreport)) 
	except:
		logger.exception("[%s] error processing", fileid)
		statusreport['status'] = 'Error processing.'
		statusreport['start'] = time.strftime('%Y-%m-%dT%H:%M:%S') 
		channel.basic_publish(exchange='',
                routing_key=header.reply_to,
                properties=pika.BasicProperties(correlation_id = \
                                                header.correlation_id),
                body=json.dumps(statusreport))		
	finally:
		statusreport['status'] = 'DONE.'
		statusreport['start'] = time.strftime('%Y-%m-%dT%H:%M:%S')
		channel.basic_publish(exchange='',
							routing_key=header.reply_to,
							properties=pika.BasicProperties(correlation_id = \
														header.correlation_id),
							body=json.dumps(statusreport))
		if inputfile is not None:
			try:
				os.remove(inputfile)
			except OSError:
				pass
			except UnboundLocalError:
				pass
Code Block
languagepy
titleCreate Word Count
def create_word_count(inputfile, ext, host, fileid):
	global logger

	(fd, inputfile)=tempfile.mkstemp(suffix='.' + ext)
	try:
		# make syscall to wc
		subprocess.check_output(['wc', inputfile], stderr=subprocess.STDOUT)

		if(os.path.getsize(wcfile) == 0):
			raise Exception("File is empty.")

		# upload word count file

Calling R Scripts from Python

...

Versus extractors serve to extract a signature from a file's content.  These signatures, effectively a hash for the data, are typically numerical vectors which capture some semantically meaningful aspect of the content so that two such signatures can then be compared using some distance measure.  Within Versus extractors operate on a data structure representing the content of a file, produced a Versus adapter, and the returned signatures compared by either a Versus similarity or distance measure.  The combination of these adapters, extractors, and measures in turn compose a comparison which can be used for relating files according their contents.

...

Code Block
languagejava
titleMeasure
public class WordCountMeasure implements Serializable,Measure {

	private static final long SLEEP = 10000;

	@Override
	public Similarity compare(Descriptor feature1, Descriptor feature2)
			throws Exception {
		Thread.sleep(SLEEP);
		return new SimilarityNumber(0);
	}

	@Override
	public SimilarityPercentage normalize(Similarity similarity) {
		return null;
	}

	@Override
	public String getFeatureType() {
		return WordCountMeasure.class.getName();
	}

	@Override
	public String getName() {
		return "Word Count Measure";
	}

	@Override
	public Class<WordCountMeasure> getType() {
		return WordCountMeasure.class;
	}
       }

            return new SimilarityNumber(Math.sqrt(sum), 0, 1, 0);
        } else {
            throw new UnsupportedTypeException();
        }
    }
}