File streaming using Java RMI

In client-server applications there's often the need to transfer files between both parties. This blog shows how to implement a file transfer using remote capable input and output streams and how to improve the performance in WAN scenarios where network connections even with good bandwidth still suffer from high latency.

Java Remote Method Invocation (RMI) is a very easy and proven way to implement client-server or process-process communication. And since Java 1.5 stubs and skeletons are created automatically using dynamic proxies without need to deal with the RMI compiler tool (rmic).

Sample Server using an embedded RMI Registry:

Sample Client implementation:

RMI works using call by value. Instances are serialized if they are not itself remote (able) objects (implement the Remote interface), in the later case a reference is transferred.

Much of the Java IO APIs use InputStreams and OutputStreams. Now how can RMI be used to allow for remote streaming? InputStream and OutputStream are abstract classes and only a few methods have to be implemented (a full implementation may add methods like skip, mark, flush, etc. as well):

• InputStream.read()
• InputStream.read(byte[] b, int off, int len)
• InputStream.close()

• OutputStream.write()
• OutputStream.write(byte[] b, int off, int len)
• OutputStream.close()

OutputStream is easy since it already complies with RMI:

Input parameters cannot be modified so InputStream.read(byte[] b, int off, int len) doesn't work. A RMI suitable variant could be:

To get back to standard InputStream and OutputStream classes, we wrap these RMI classes into serializable classes:

For the users, these classes look like and act like standard input and output streams. Well, they may throw RemoteExeptions, but luckily RemoteExeptions are derived from IOExceptions which have to be handled anyway.

RMIInputStreamInterf and RMIOutputStreamImpl wrap given input and output streams and export themselves as remote objects:

Now we can add two methods to our Server and SeverImpl:

Sample to use it:

Notes

• The concept even allows for daisy-chaining: client requests a stream from server 1 which in turn requests the stream from server 2.
• For on the fly data compression you can insert the deflater/inflater stream classes from java.util.zip (since Java 1.6 the implementation is symmetrical providing deflaters and inflaters for both input and output streams).
• UnicastRemoteObject.exportObject(this, 1099) is worth mentioning for two things: It's possible to export objects on the same port as the RMI registry which limits the number of holes to punch through firewalls. And it automatically creates proxy stub classes which UnicastRemoteObject.exportObject(this) wouldn't.

Still for file transfer the implementation has a drawback: The copy method reads and writes blocks and either call is a remote call. On a LAN that's no big issue, but with ADSL-connections we have latencies of around 50ms or more. Even with infinite bandwidth, to transfer 100MB using 64 KB blocks would cost us 80s assuming a latency of 50ms. A larger block size improves the performance, but we shouldn't waste memory.

The trick to improve performance is to perform the file transfer "inline" as part of the RMI serialization.

RMIPipe is a serializable object that during serialization transfers data from an input to an output stream. RMIPipe does its own serialization by overriding the writeObject and readObject methods. The serialization reads data from an InputStream instead of serializing some class variables, the deserialization writes data to an OuputStream.

Actually it's two implementations in one. Initialized on a (local) OutputStream, it will register the OutputStream locally in a static registry. The registration key will be delivered to the remote side. The remote side (InputStream side) then creates an appropriate RMIPipe object and sends it back. Initialized on a (local) InputStream, it will remember the stream and serialize it if the RMIPipe instance is serialized.

Adding to RMInputStreamImpl:

Adding to RMIOutputStreamImpl:

Adding to RMIInputStream:

Adding to RMIOutputStream:

The optimized copy method of the TestClient now looks like: