Here is a potential solution which should work without any changes to the Encoder interface.

In your Encoder, if you declare a *public *method called setParent then Joran will inject the enclosing appender. If the enclosing appender is an instance  of FileAppender, then your Encoder can invoke the isAppend() and getOutputStream() method of the enclosing FileAppender instance from within Encoder.headerBytes().

At the point where Encoder.headerBytes() is invoked, the file is already opened and all fields of the enclosing FileAppender have been initialized.