...
Notes for implementing our own regular expression engine
- Interesting: http://matt.might.net/articles/implementation-of-regular-expression-matching-in-scheme-with-derivatives/
- A follow-on post: A non-blocking lexing toolkit for Scala in less than 800 lines of code, from regex derivatives (http://matt.might.net/articles/nonblocking-lexing-toolkit-based-on-regex-derivatives/)
- A great series of articles by Russ Cox: Implementing Regular Expressions (http://swtch.com/~rsc/regexp/)
Regular Expression Matching Can Be Simple And Fast (but is slow in Java, Perl, PHP, Python, Ruby, ...) (http://swtch.com/~rsc/regexp/regexp1.html) - Introduces several implementation techniques
Regular Expression Matching: the Virtual Machine Approach (http://swtch.com/~rsc/regexp/regexp2.html) - Discusses several implementation details, including a section on POSIX longest leftmost matching. Includes some links to some test suites for POSIX matching rules.
Regular Expression Matching in the Wild (http://swtch.com/~rsc/regexp/regexp3.html) - Tour of the re2 library (written in C++) - useful for translating the implementation to Java or Scala (which no one has apparently done yet)
...
- Someone started a binary parser and pickler combinator library. https://github.com/teigen/byteme Still looks immature, but might provide some good ideas.
Miscellaneous notes:
java scanner - takes Readable as input (good), doesn't allow regex that matches at beginning of stream.
Implement longest match by rotating through mulitiple regex matchers, which determine if the pattern matches immediately at the beginning of the supplied characters.
Java regex Matcher class takes only CharSequence as input.
CharSequence is finite (has length() method) which precludes (most likely) creating a virtualized stream-like behavior underneath our own CharSequence implementation.
---------------------
Scala ||| combinator of regex parsers produces longest match to the combined regex's.
Scala combinator parsers read from a Reader.
The reader can virtualize the character stream up until a decode error, or size limit of our choice, and can hide all details of when additional blocks of data must be retrieved.
val rdr = new InputStreamReader(Channels.newInputStream(rbc : ReadableByteChannel)
// Must figure out what happens if character decoding gets an error.
// Worst case, we have to implement Reader interface ourselves with code that calls the decoder.
// length limit on delimiter match could be implemented by creating a finites substream from the rbc, instead of just handing the original rbc to it.
val pr = Parser.parse(rdr)