Using FilterIO for wrapping a write stream

[wr0ngway]

Is there any reason you didn't implement write capability into FilterIO? It seems like it would be useful to define filters for either direction on a stream. I'd be happy to take a crack at it if you could give me some pointers as to how you'd like to see it work.

[jasoncodes]

The only reason is that I haven’t had a use case for this yet. Did you have a use case in mind?

It seems to me that perhaps this may not even be required. Here’s a quick bit of code that filters stdout:

read_io, write_io = IO.pipe
filtered_io = FilterIO.new(read_io) { |data| data.upcase }
thread = Thread.new { IO.copy_stream filtered_io, $stdout }
write_io.puts 'test'
write_io.close
thread.join

[wr0ngway]

That could work, but requires one to use a thread, when really it shouldn't be necessary.

The use case I was thinking of was for a class which provides an open method similar to Kernel.open, which returns an IO object. I need to be able to wrap that IO object with additional behavior, and pass it around, e.g. In my use case, a filtered stream need only be read OR write, not both at once, but the example below would also work for a R/W stream (Assuming FilterIO had a write).

class Foo
  def open(file, mode)
    return open(file, mode)
  end
end

class DumbEncrypt < SimpleDelgator
  def open(file, mode)
    FilterIO.new(super(file, mode)) do |data, state|
      data.reverse
    end
  end
end

def factory
  obj = Foo.new
  # factory wraps original in many, variable decorators depending on usage, just one here for brevity
  wrapped = DumbEncrypt.new(obj)
end

io = factory.open('foo.txt', 'w')
io.write("hello")
io.close
# foo.txt is olleh

io = factory.open('foo.txt', 'r')
data = io.read # 'hello'
io.close

[wr0ngway]

I'm thinking something like this would do it, though I'm not sure about the encoding stuff nor the BlockState - eof is always false as we can't detect it, bof can only be detected by maintaining state internal to FilterIO or by checking the @io.pos which is sensitive to seeking:

def write(data)
  data = data.dup
  if data
    data.encode! internal_encoding if internal_encoding
    data.force_encoding external_encoding if external_encoding
  end

  args = [data]
  args << BlockState.new(@io.pos == 0, false) if @block.arity > 1
  @io.write(@block.call(*args))
end

[jasoncodes]

The code you present here looks like it’s on the right track.

Requesting of more data is definitely important for a bunch of transformations so that needs to be supported. BOF would be nice. EOF I think can safely stay false as it would only be true after close.

I am concerned about using a single FilterIO for both read and write. It seems like there’s way too many scenarios to consider. As such, I think this should be unsupported and there should be protections against FilterIO being used this way. Perhaps this could be implemented as guards in all public methods which exist for reading or writing but not both?

I am wondering if the code may be cleaner and more maintainable if write support were in a separate class entirely. Reasoning is the new class could just have the methods appropriate for writing and also the buffering logic is likely to be significantly different.

[wr0ngway]

I'm not sure how one would request more data as part of the IO write API since the client is driving wether to put more data in there or not (i.e. they call write as often as they like before calling close). For a client of FilterIO, buffering for the purposes of chunking/batching data could be done this way (i.e. leave it up to the client):

def open
  write_buffer = '' # local lexical state preserved by block
  io = FilterIO.new(dest_io) do |data, state|
    write_buffer << data
    chunk = write_buffer.slice!(0, 10) # mutating buffer, taking only what code needs
    output = process(chunk)
    output
  end
  return io
end

We could also have buffering for writes built into FilterIO, but would have to flush through the block on close. However, that would necessitate having api for buffer management within the block (i.e. a way to say I'm only using some of the data, and leaving the rest for the next pass through). Might just be easier to leave that up to the client.

For the state, bof? works as shown in my previous comment, eof? could return @io.closed? to be safe.

For R/W streams, we could add a direction (== :read or :write or reading? writing?) to the state block param, though I'm ok with R/W being unsupported too - A separate class or guards could also work. I'm ambivalent except that to solve my problem, the shortest path is to just add a write() method to the single class :)

How about the encode stuff, does it even need to be there, or should it just rely on the destination io for handling encoding?

[jasoncodes]

I was picturing the requesting more data API to work exactly the same as reading (raising NeedMoreData, returning unused data back to the buffer, etc.). The buffering required for this is an implementation detail that I do not think the user of FilterIO should need to know about nor re-write over and over. The buffer would indeed have to be flushed at close. There would be one final call to the block with eof? returning true. I think this is important to implement.

Keeping everything in a single class with guards to prevent read/write direction switching would probably be the cleanest API. I am picturing the direction initially being nil and the guards would call a helper method with the desired direction. First call sets it and every other call verifies. At minimum I think write and puts should be implemented. All other methods (read, gets, pos, etc.) would be guarded as read-only methods. The guard should be on all public methods.

I am wondering if the easiest and cleanest way to go about this though would be some kind of delegation. Move the current code into an internal read class and delegate all calls through from the main class. A new internal write class would be created which implements writing. The internal instance would be instantiated on first read/write and no guards would be required. You just get a NoMethodError if you call something unavailable for that direction.

As for encodings, we’d want to pass the block the data in the internal encoding of the stream. The target stream can do the conversion to the external encoding. I don’t think we require any explicit code to do this.

[jirutka]

+1

[stevecj]

I have a specific use case for filtering a write stream. I'd like write data using Zlib::GzipWriter and have the output of that be ASCII85-encoded while it is being written out to a file.