After porting my mod_mono ASP.NET MVC application to Ruby and Rails and setting up Phusion Passenger up to run the application under mono, I finally figured out how to fix the leaking semaphore issue. The real title of this post should probably be "PEBKAC or Don't assume errors are unrelated, you idiot".
The problem manifests itself as a build up of semaphore arrays by the apache process, which is visible via ipcs. When the site is first started the output looks like this:
At some point all ASP.NET pages will return blank pages. No errors, no nothing, .NET logging reports normal behavior, but no content is sent. And you can restart the mono processes and apache all you want, it won't come back. Sorry.
Since day one i'd been receiving warnings at apache startup up, but since i didn't understand what they meant and things seemed to be working, i had been ignoring the,. Of course, that was a lie on its face. Things were clearly not working, with the leaking semaphores, but I conveniently filed the two issues as unrelated in my head and ignored them at my peril. The warning was this:
[Mon Jan 24 00:12:50 2011] [crit] The unix daemon module not initialized yet.
Please make sure that your mod_mono module is loaded after the User/Group
directives have been parsed. Not initializing the dashboard.
This warning was repeated for as many times as I had ASP.NET vhosts defined. I looked at my vhost configurations and saw nothing about users and groups and thought it was some weird mono issue and left it at that. But the actual problem was not in the vhost configuration but in httpd.conf. The problem was this default section:
#
# Load config files from the config directory "/etc/httpd/conf.d".
#
Include conf.d/\*.conf
#
# If you wish httpd to run as a different user or group, you must run
# httpd as root initially and it will switch.
#
# User/Group: The name (or #number) of the user/group to run httpd as.
# . On SCO (ODT 3) use "User nouser" and "Group nogroup".
# . On HPUX you may not be able to use shared memory as nobody, and the
# suggested workaround is to create a user www and use that user.
# NOTE that some kernels refuse to setgid(Group) or semctl(IPC_SET)
# when the value of (unsigned)Group is above 60000;
# don't use Group #-1 on these systems!
#
User apache
Group apache
Obviously User and Group are set after all vhost configs are loaded. Pretty much exactly what the warning was saying (doh). I simply moved the Include below User/Group and since then I have not seen more than 9 semaphores and I've restarted mono, rebuilt the application and hit the app with ApacheBench, the combination of which used to drive semaphores up.
Since things are working now and ASP.NET MVC under mod_mono is significantly faster than the Rails port, I'm sticking with ASP.NET MVC for production right now, monitoring semaphores to make sure this really did fix the problem.
For notify.me I hand-rolled a simple actor system to handle all Xmpp traffic. Every user in the system has its own actor that maintains their xmpp state, tracking online status, resources, resource capability, notification queues and command capabilities. When a message comes in either via our internal notification queues or from the user, a simple dispatcher sends the message on to the actor which handles the message and responds via a message that the dispatcher either hands off to the Xmpp bot for formatting and delivery to the client or sends it to our internal queues for propagation to other parts of the notify.me system.
This has worked flawlessly for over 2 years now, but its ad-hoc nature means it's a fairly high touch system in terms of extensibility. This has led me down building a more general actor system. Originally Xmpp was our backbone transport among actors in the notify.me system, but at this point, I would like to use Xmpp only as an edge transport, and otherwise use in-process mailboxes and serialize via protobuf for remote actors. I still love the Xmpp model for distributing work, since nodes can just come up anywhere, sign into a chatroom and report for work. You get broadcast, online monitoring, point-to-point messaging, etc. all for free. But it means all messages go across the xmpp backbone, which has a bit of overhead and with thousands of actors, i'd rather stay in process when possible. No point going out to the xmpp server and back just to talk to the actor next to you. I will likely still use Xmpp for Actor Host nodes to discover each other, but the actual inter-node communication will be direct Http-RPC (no, it's not RESTful, if it's just messaging).
One design approach I'm currently playing with is using actors that expose their contract via an interface. Keeping the share-nothing philosophy of traditional actors, you still won't have a reference to an actor, but since you know its type, you know exactly what capabilities it has. That means rather than having a single receive point on the actor and making it responsible for routing the message internally based on message type (a capability that lends itself better to composition), messages can arrive directly at their endpoints by signature. Another benefit is that testing the actor behavior is separate from its routing rules.
While this is simple and decoupled, it is implictly synchronous. Sure .Notify could be considered a fire-and-forget message, .QueryStatus definitely blocks. And if we wanted to communicate an error condition like not finding the recipient, we'd have to do it as an exception, moving errors into the synchronous pipeline as well. In order to retain the flexibility of a pure message architecture, we need a result handle that let's us handle results and/or errors via continuation.
My first pass at an API for this resulted in this calling convention:
_mailbox.Send<IXmppAgent>("foo@bar.com",a => a.Notify("hey", "how'd you like that?"));
var result = _mailbox.SendAndReceive<IXmppAgent, OnlineStatus>(
"foo@bar.com",
a => a.QueryStatus()
);
I'm using MindTouch Dream's Result<T> class here, instead of Task<T>, primarily because it's battle tested and I have not properly tested Task under mono yet, which is where this code has to run. In this API, .Send is meant for fire-and-forget style messaging while .SendAndReceive provides a result handle -- and if Void were an actual Type, we could have dispensed with the overload. The result handle has the benefit of letting us choose how we want to deal with the asynchronous response. We could simply block:
var status = _mailbox.SendAndReceive<IXmppAgent, OnlineStatus>(
"foo@bar.com",
a => a.QueryStatus())
.Wait();
Console.WriteLine("foo@bar.com status:", status);
or we could attach a continuation to handle it out of band of the current execution flow:
_mailbox.SendAndReceive<IXmppAgent, OnlineStatus>(
"foo@bar.com",
a => a.QueryStatus()
)
.WhenDone(r => {
var status = r.Value;
Console.WriteLine("foo@bar.com status:", status);
});
or we could simply suspend our current execution flow, by invoking it from a coroutine:
var status = OnlineStatus.Offline;
yield return _mailbox.SendAndReceive<IXmppAgent, OnlineStatus>(
"foo@bar.com",
a => a.QueryStatus()
)
.Set(x => status = x);
Console.WriteLine("foo@bar.com status:", status);
Regardless of completion strategy, we have decoupled the handling of the result and error conditions from the message recipient's behavior, which is the true goal of the message passing decoupling of the actor system.
Looking at the signatures there are two things we can still improve:
If we send a lot of messages to the same recipient, the syntax is a bit repetitive and verbose
Because we need to specify the recipient type, we also have to specify the return value type, even though it should be inferable
We can address both of these, by providing a factory method for a typed mailbox:
public interface IMailbox {
IMailbox<TRecipient> To<TRecipient>(string id);
}
public interface IMailbox<TRecipient> {
void Send(Expression<Action<TRecipient>> message);
Result SendAndReceive<TResponse>(Expression<Action<TRecipient>> message);
Result<TResponse> SendAndReceive<TResponse>(
Expression<Func<TRecipient, TResponse>> message
);
}
which let's us change our messaging to:
var actorMailbox = _mailbox.To<IXmppAgent>("foo@bar.com");
actorMailbox.Send(a => a.Notify("hey", "how'd you like that?"));
var result2 = actorMailbox.SendAndReceive(a => a.QueryStatus());
// or inline
_mailbox.To<IXmppAgent>("foo@bar.com")
.Send(a => a.Notify("hey", "how'd you like that?"));
var result3 = _mailbox.To<IXmppAgent>("foo@bar.com")
.SendAndReceive(a => a.QueryStatus());
I've included the inline version because it is still more compact than the explicit version, since it can infer the result type.
The reason the mailbox uses Expression instead of raw Action and Func is that at any point an actor we're sending a message to could be remote. The moment we cross process boundaries, we need to serialize the message. That means we need to be able to programatically inspect the inspection, and build a serializable AST as well as serialize the captured data members used in the expression.
Since we're talking serializing, inspecting the expression also allows us to verify that all members are immutable. For value types, this is easy enough, but DTOs would need be prevented from changing so that local vs. remote invocation won't end up with different result just because the sender changed it's copy. We could handle this via serialization at message send time, although this looks like a perfect place to see how well the Freezable pattern works.
I've been using Apache with mod_mono for some ASP.NET MVC2 projects and kept having problems with semaphore arrays being leaked. Under 2.6.7 this even broke xbuild after a while. I then went to 2.8 and 2.8.1, but it didn't stop the leaks. I posted on the mono-devel list and after lack of response simply asked if anyone was actually running ASP.NET under mod_mono, which also elicited no replies. Finally, I posted the problem on stackoverflow, again without any resolution.
The problem manifests itself as a build up of semaphore arrays by the apache process, which is visible via ipcs. When the site is first started the output looks like this:
At some point all ASP.NET pages will return blank. No errors, no nothing, .NET logging reports normal behavior, but no content is sent. And you can restart the mono processes and apache all you want, it won't come back. Sorry.
What does work is to remove all semaphore arrays via ipcrm and restart apache. For the time being i've had a script in cron that did this:
/usr/bin/ipcrm sem $(/usr/bin/ipcs -s | grep apache | awk '{print$2}');
Unfortunately, the leaking semaphores are somehow related to traffic, so eventually i'd either have to increase the frequency of the restart script or make it more intelligent. I opted for neither and decided to try out nginx+fastcgi+mono.
Like my mono 2.8.1 install, I'm doing this on an Amazon Linux AMI 1.0. And like that article, this isn't so much a recipe than a log of my actions. Note that this was done after the 2.8.1 install from source so there might be dependencies i'm not mentioning since they'd already been addressed.
Voila, ASP.NET MVC2 under nginx. This may have other issues, but i have not yet observed them, so this seems to be a way to get around the mod_mono issues.
Of course that's a bit cumbersome. What we really need is an init script so we can start and stop teh fastcgi server like other services:
#!/bin/sh
# chkconfig: - 85 15
# description: Fast CGI mono server
# processname: fastcgi-mono-server2.exe
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/mono-2.8.1/bin
DESC=fastcgi-mono-server2
WEBAPPS="/:/foobar/http/www/"
LISTENER="tcp:127.0.0.1:9000"
MONOSERVER=/opt/mono-2.8.1/bin/fastcgi-mono-server2
MONOSERVER_PID=$(ps auxf | grep "${LISTENER}" | grep -v grep | awk '{print $2}')
case "$1" in
start)
if [ -z "${MONOSERVER_PID}" ]; then
echo "starting mono server"
${MONOSERVER} /applications=${WEBAPPS} /socket=${LISTENER} &
echo "mono server started"
else
echo ${WEBAPPS}
echo "mono server is running"
fi
;;
stop)
if [ -n "${MONOSERVER_PID}" ]; then
kill ${MONOSERVER_PID}
echo "mono server stopped"
else
echo "mono server is not running"
fi
;;
esac
exit 0
While this takes care of my ASP.NET troubles, it now means that I'd have to migrate the various php packages over as well. Wordpress is no problem, but OpenCart would be a bit of hacking, which is really the last thing I want to do when it comes to ecom.I thought about running both nginx and apache and using one to proxy the sites on the other (since EC2 won't let me attach multiple IPs to a single host), but decided against that as well, since it would just be a hack of a different color. There's also the option of running fastcgi against apache, but I've not found any docs on how to set up ASP.NET MVC that way, all the existing examples map ASP.NET file extensions to fastcgi, which isn't an option.
Apache is still the most supported solution, so when integrating a number of sites on a single host, it ends up being the best option. It's just that mod_mono doesn't seem to be playing along for me :( So, I hatched a scheme to rid myself of ASP.NET for this site, since it really only has trivial business logic and I have a holiday coming up. More on that later.
Since last year ended up being an academic exercise in learning javascript, that task is still outstanding. But it is no less important. I consider javascript to be the one language I cannot afford not being good at. It is truly the assembly language of the web and is rapidly gaining ground on the server as well. I may or may not warm up to going back to a dynamically typed language but that's really independent of the need represented by this gap in expertise.
In many ways, javascript provides a lot of the features I am aiming for with promise. It follows the same pattern of methods being lambda's attached to slots on objects. Although syntax-wise, CoffeeScript is an even better match -- getting rid of the overly verbose function prefix for lambda's among many other cool changes.
I still doubt that javascript will become my new favorite language, simply because of my strongly-typed tendencies. But being a C# programmer i'm in kind of a weird space. I don't much care for windows as client and abhor it as a server. So almost everything I do in C# runs under mono. I admire Miguel DeIcaza's relentless drive for creating the best environment possible regardless of detractors and am continually amazed at the quality and completeness of mono. That said being a C# advocate in the linux world is asking for additional pain. Mono will always be trailing MS's implementation by a bit and for your troubles you do end up being a pariah in the linux world. Finding a language that is more at home and accepted on my favorite platform would be beneficial. For a long time, java and C# were just close enough that in the worst case scenario, I could always go java again. But now that i'm used to C#'s lambda syntax and linq, java just feels ancient and dead to me.
Of all the languages I've looked at Scala hits my personal feature bingo the best. I love the actor pattern and built it using Xmpp in C# for notify.me. From my sideline review Akka seems to be the actor implementation to beat, so picking a project and implementing it start to finish in Scala/Akka seems like the way to go. After that I should have enough of a feel for the language to see whether it's a contender for my C# affections
To stick with the common thread, I think that going forward in the mobile space, javascript once again is going to be the most important tool in the development toolkit. But at the same time, I am a sucker for native clients and am happy that the current crop of smartphones have revived writing client side software.
However, the last time I did mobile client programming was WM5, so I have some serious catching up to do. The goal here is to pick a useful app and write it for all three of the above platforms and release it. I'm going to stick somewhat to my comfort zone here by using C#, the default on Windows Phone 7 and enabled by MonoTouch and MonoDroid on the other two. The departure from my comfort zone is venturing back to the client after spending almost all my time on the server and figuring out that the re-use vs. platform specific stories are and what deployment looks like. I've not settled on an app, but most likely it will be a native notify.me client.
Those resolutions should keep me busy enough, especially since they are spare time activities for when I'm not busy working on MindTouch and Dream, extending notify.me or maintaining curdsandwine.com.
I generally stay away from anything that's not in rpm form, because playing the configuration/build game is just such a waste of time especially when trying to come up with a repeatable recipe. Unfortunately, there's no rpm's for 2.8.x, 2.6.7 being the latest on the mono-project site.
I don't have a clue about building rpm's from source and have not found a build recipe either, but if one exists, I'd be more than happy to start building and distributing current rpm's for mono, since i don't feel like building from source on every new machine and having that machine's config fall out of the management purview of yum/rpm.
The purpose of build 2.8.1 is running ASP.NET MVC2. In theory this should work under 2.6.7, but there's a couple of bugs that mean it's not quite usable for production purposes. While I develop in Visual Studio, i don't want to commit binaries to my git repo and I want the ability to make fixes with emacs on the development server. To make this work, I also include xbuild as part of the recipe, since it lets me build the Visual Studio solution.
Anyway, here's the chronicle of my install. This is less of an article than a log of what I did to get mono 2.8.1 installed from source on a fresh Amazon Linux AMI 1.0, so it may include things you don't need that were dead-ends i went down. I started by looking at Nathan Bridgewater's Ubuntu & Fedora recipe.
cd /tmp
mkdir mono-2.8.1
cd mono 2.8.1
wget http://ftp.novell.com/pub/mono/sources/xsp/xsp-2.8.1.tar.bz2
wget http://ftp.novell.com/pub/mono/sources/mod_mono/mod_mono-2.8.tar.bz2
wget http://ftp.novell.com/pub/mono/sources/mono/mono-2.8.1.tar.bz2
wget http://ftp.novell.com/pub/mono/sources/libgdiplus/libgdiplus-2.8.1.tar.bz2
tar jxf libgdiplus-2.8.1.tar.bz2
tar jxf mod_mono-2.8.tar.bz2
tar jxf mono-2.8.1.tar.bz2
tar jxf xsp-2.8.1.tar.bz2
cd libgdiplus-2.8.1
./configure --prefix=/opt/mono-2.8.1 (30s)
make
make install
cd ..
cd mono-2.8.1
./configure --prefix=/opt/mono-2.8.1 (2m)
make
make install
cd ..
In order to build xsp, we need to configure the environment to find mono:
I did this build on an Amazon EC2 micro instance, and it took 3 hours. So might be worth attaching the EBS to a EC2 High CPU instance building it there and then re-attaching it to a micro. But you can see why I prefer rpm's. Sure i can zip up the build directories and run make install on another identical machine, but the make install on the micro instance for mono took 20 minutes. Compare that to the 2.6.7 rpm install, which took me 42 seconds including downloading via yum from novell.
A couple of years back I wrote a store-and-forward message queue called simpleMQ for vmix. A year later, Vmix was kind enough to let me open source it and I put it up on sourceforge (cause that was the place back in the day). But it never got any documentation or promotion love from me, being much too busy building notify.me and using simpleMQ as its messaging backbone. Over those last couple of years, simpleMQ was served us incredibly well at notify.me, passing what must be billions of messages by now. But it does have warts and I have a backlog of fixes/features i've been meaning to implement.
Rather than invest more time on simpleMQ, i've been watching other message/work queues to see whether there is another product i could use instead. I've yet to find a product that i truly like, but Beanstalkd is my favorite of the bunch. Very simple, fast and with optional persistence, it addresses most of my needs.
Beanstalkd's protocol is inspired by memcached. It uses a persistent TCP connection, but relies on connection state only to determine which "tube" (read: workqueue) you are using and to act as a work timeout safeguard. The protocol is ASCII verbs with binary payloads and uses yaml for structured responses.
Tubes are created on demand and destroyed once empty. By default beanstalkd is in-memory only, but can use a binary log to store items and recover the in-memory state by log playback. I had briefly looked at zeromq, but after finding out that its speed relies on no persistence, I decided to give it a skip. zeromq might be web scale, but i prefer a queue that doesn't degrade to behaving like /dev/null :) Maybe my transactional RDBMS roots are showing, but I have a soft spot for at least journaling data to disk.
One concept of beanstalkd that i'm still conflicted about is that work is given a processing time-out (time-to-run) by the producer of the work, rather than having the consumer of the work declare its intended responsiveness. Since the producer doesn't know how soon the work gets picked up, i.e. time-to-run is not a measure of work item staleness, I don't see a great benefit for having the producer dictate the work terms.
The other aspect of work distribution beanstalkd lacks for my taste is the idea of being able to produce work in one instance and have it forwarded to another instance when that instance is available, i.e. store-and-forward queues. I like to keep my queues on the current host so i can produce work without having to rely on the uptime of the consumer or some central facility. However, store-and-forward is an implementation detail I can easily fake with a daemon on each machine that acts as a local consumer and distributor of work items, so it's not something i hold against beanstalkd.
Notify.me being a mix of perl and C#, i needed a .NET client. A protocol complete one not existing and given the simplicity of the Beanstalkd protocol, I opted to write my own and have released it under Apache 2.0 on github.
I've not put DLLs up for downlad, since the API is still somewhat in flux as I continue to add features, but the current release supports the entire 1.3 protocol. By default it considers all payloads as binary streams, but I've included extension methods to handle simple string payloads:
// connect to beanstalkd
using(var client = new BeanstalkClient(host, port)) {
// put some data
var put = client.Put("foo");
// reserve data from queue
var reserve = client.Reserve();
Console.Writeline("data: {0}", reserve.Data);
// delete reserved data from queue
client.Delete(reserve.JobId);
}
The binary surface is just as simple:
// connect to beanstalkd
using(var client = new BeanstalkClient(host, port)) {
// put some data
var put = client.Put(100, 0, 120, data, data.Length);
// reserve data from queue
var reserve = client.Reserve();
// delete reserved data from queue
client.Delete(reserve.JobId);
}
I've tried to keep the interface IBeanstalkClient to be a close as possible to the protocol verb signatures and rely on extension methods to create simpler versions on top of that interface. To facilitate extensions that provide smart defaults, the client also has an instance of a Defaults member that can be used to initialize those values.
The main deviation from the protocol is how I handle producer and consumer tubes. Rather than have a separate getter and setter for the tube that put will enter work into, I simply have a settable property CurrentTube. And rather than surfacing watch, ignore and listing of consumer tubes, the client includes a special collection, WatchedTubes, with the following interface:
I was originally going to use ICollection<string>, but Clear() did not make sense and I wanted to have a manual method to reload the list from the server, which is exposed via Refresh(). Under the hood, watched tubes is a hashset, so adding the same tube multiple times has no effect, neither is order of tubes in the collection guaranteed.
The client is functional and can do everything that Beanstalkd offers, but it's really just a wire protocol, akin to dealing with files as stream. To make this a useful API, it really needs to take the 90% use cases and remove any friction and repetition they would encounter.
BeanstalkClient isn't, nor is it meant to be, thread-safe. It assumes you create a client when you need it and govern access to it, rather than sharing a single instance. This was motivated by Beanstalkd's behavior of storing tube state as part of the connection. Given that I encourage clients to be created on the fly to enqueue work, it makes sense that under the hood clients should use a connection pool both to re-use existing connections rather than constantly open and close sockets and to limit the maximum sockets a single process tries to open to Beanstalkd. Pooling wouldn't mean sharing of a connection by clients, but handing off connections to new clients and putting them in a pool to be closed on idle timeout once the client is disposed.
Most of this work is complete and on a branch, but i want to put it through some more testing before merging it back to master, especially since it will introduce client API changes.
Does beanstalk inherently support distributed servers?
Yes, although this is handled by the clients, just as with memcached. The beanstalkd server doesn't know anything about other beanstalkd instances that are running.
I need to take a look at the clients that do implement this and determine what that means for me. I.e. do they use some kind of consistent hashing to determine which node to use for a particular tube, etc. But I do want to have parity with other clients on this.
For me, the 90% use case for a work queue is produce work on some threads/processes/machines and consume that work on a number of workers. Generally that item will have some structured fields describing the work to be done and producers and consumers will use designated tubes for specific types of work. These use cases imply that producers and consumers are separate user stories, that they are tied to specific tubes and deal with structured data. My current plan is to address these user stories with two new interfaces that will look similar to these:
The idea with each is that it's tied to a tube (or tubes for the consumer) at construction time and that the implementation will have a simple way of associating a serializer for the entity T (will provide protobuf and MetSys.Little support out of the box).
Once there is the concept of a Job<T>, it makes sense that reservation of jobs should be exposed as a stream of work that can be processed via link. Although since items should only be reserved when the subscriber accepts the work, it should probably be encapsulated in something like this:
This way, multiple subscribers can try to reserve an item and items not reserved by anyone are released automatically.
As I work on the future work items, I will also use the library in production so i can get better educated about the real world behavior of Beanstalkd and what uncovered scenarios the client runs into. There is ok test coverage over the provided behavior but I certainly want to increase that signficantly as i keep working on it.
For the time being, I hope the library proves useful to other .NET developers and would love to get feedback, contributions and issues you may encounter.
As I talked about recently, the standard binaries for db4o have some issues with mono, so I recompiled the unmodified source with the MONO configuration flag. I've packed up both the debug and release binaries and you can get them here. These are just the binaries (plus license). It's not the full db4o package. If you want the full package, just get it directly from the db4o site, since the MONO config flag and have Visual Studio rebuild the package.
In a world that is increasingly green, it is a waste of perfectly healthy computer cycles to interpret your code when you can use an optimizing JIT compiler to run your code.
I'm going to presume that's with tongue firmly implanted in cheek.
Miguel de Icaza just posted a long entry on a Hackathon the mono team did in the last 21 days to get some version of Moonlight ready for Remix 07 and it looks like they succeeded. Further progress information can also be found on the Moonlight site.
Next to the Silverlight 1.1 alpha which exposes the CLR, Moonlight is probably one of my favorite tech project to follow at this time. I don't even run X anywhere anymore, just using linux for servers, so I don't think i'll ever need Moonlight. But its the availability that has me excited. I am personally rooting for a web programming model that doesn't force javascript on you for client side programming, but I don't think Win/MacOS is "cross-platform" enough to establish this as a norm, rather than a specialty plug-in. I hope Moonlight tips the adoption likelihood in Silverlight's favor.
Currently, I'm mostly waiting for a Socket API to make it into Silverlight 1.1. Once that's available, I'm going to make sure that my LFSLib can run under silverlight. This would open the door for some very cool in-brower LFS admin tools. I also have an internal DirectX implementation of the LFS smx and pth formats for rendering top down views. I've been looking at porting that to WPF, since i don't required 3D and just rendering a plane at this point. If I can reduce it further to the Silverlight Xaml subset, I could easily produce a Silverlight equivalent of LFS Spectator. But I'm getting ahead of myself. Right now I just need to get full patch X support out the door for LFSLib.NET. The protocol is complete now, I just need to complete the TCP code, do some doc clean-up and testing.
I was just trying to move a remoting service from .NET 2.0 to mono and ran into problems with Interop. The client & server would work fine if running on the same architecture, but fail when crossing architectures. Turns out that the secure portion of TcpChannels isn't completed yet on mono, throwing an AuthenticationException when a Windows client tried to connect to a Linux server. The culprit is ISecurableChannel. To avoid this, simply, don't use the security feature for now (i.e. use at own risk, best on a private net).
