________________________________________
From: Stephen Lawrence
Sent: Wednesday, May 04, 2016 2:02 PM
To: Mike Beckerle
Cc: Stephen Lawrence
Subject: [PATCH 1/1] Fix unparsing thread leak

From: Pat McClory <pat@quarksecurity.com>

When unparsing XML the producer thread of the Coroutine calls setNext(), which unblocks
the consumer thread, and then it blocks:

val res = waitForResumeAny // blocks until it is resumed

The consumer (the unparser), never unblocks this thread.
So as more and more files are run through the unparser more and more blocked
threads are created. Eventually on Linux this leaves the system in an unusable
state (as the kernel can no longer allocate more pids)...

check at /proc/pid/task/ while running many files through the unparser, without
this patch it will constantly grow throughout the execution, with it it stays
constant.

Fix is to just let the producer finish "producing" so the producer's body()
method returns, and the thread exits. Maybe could short-circuit this.
—
.../main/scala/edu/illinois/ncsa/daffodil/processors/Runtime.scala | 1 +
.../edu/illinois/ncsa/daffodil/processors/unparsers/UState.scala | 6 ++++++
2 files changed, 7 insertions

diff --git a/daffodil-runtime1/src/main/scala/edu/illinois/ncsa/daffodil/processors/Runtime.scala b/daffodil-runtime1/src/main/scala/edu/illinois/ncsa/daffodil/processors/Runtime.scala
index 83e9b98..0aa364a 100644
— a/daffodil-runtime1/src/main/scala/edu/illinois/ncsa/daffodil/processors/Runtime.scala
+++ b/daffodil-runtime1/src/main/scala/edu/illinois/ncsa/daffodil/processors/Runtime.scala
@@ -308,6 +308,7 @@ class DataProcessor(val ssrd: SchemaSetRuntimeData)
unparserState.dataProc.get.init(ssrd.unparser)
unparse(unparserState)
unparserState.evalSuspendedExpressions() // handles outputValueCalc that were suspended due to forward references.
+ unparserState.infosetFlush
unparserState.unparseResult
} catch {
case ue: UnparseError =>

{ diff --git a/daffodil-runtime1/src/main/scala/edu/illinois/ncsa/daffodil/processors/unparsers/UState.scala b/daffodil-runtime1/src/main/scala/edu/illinois/ncsa/daffodil/processors/unparsers/UState.scala index f3a3f24..36e3e1f 100644 --- a/daffodil-runtime1/src/main/scala/edu/illinois/ncsa/daffodil/processors/unparsers/UState.scala +++ b/daffodil-runtime1/src/main/scala/edu/illinois/ncsa/daffodil/processors/unparsers/UState.scala @@ -143,6 +143,12 @@ class UState( m.get }

+ def infosetFlush = {
+ while (advance)

{ + advanceAccessor + }

+ }
+
def isInspectArrayEnd = {
if (!inspect) false
else {
–
2.5.5