Its Been Fun

Previously...

• IT ACTUALLY WORKED!
• I talked a bit about gen_tcp and active mode.
• Initial results were very disappointing.

I ran across an issue where ECM would only deliver one packet per blob of data given to it.  This works fine if the data blog only contains one megaphone packet, but when the server gets busy, more than one packet can be contained in each blob.  I fixed this problem --- now ECM keeps sending out data until there is nothing left to deliver.  

In addition, megaphone now sends the HTTP status code along with the rest of the data along to ECM.  ECM, in turn, uses that code when it sends a response to a client.

There is still an issue where connections get dropped during high load periods.  In addition, once the error state is reached, I cannot seem to log on using JWChat.  All this leads me to believe there is some problem lurking in ECM.

At this point, I am going to try contacting some people to see if there is an easy solution (HA!) to ejabberd's memory consumption.  Unless ejabberd's memory use can be curtailed, I don't see a lot of value in megaphone --- yes it does bring some value, but not enough to accomplish the goal that I have set (being able to host 1,000,000 users).

In any case, I will put the code for megaphone/ECM up on github in the next few days.  I may also collect some of the insights I have gained during this process.  

Next time: code and insights?

This is Not the Result You Are Looking for

Previously...

• I changed the pass-through from single to multi-threaded.
• IT ACTUALLY WORKED!
• I talked a bit about gen_tcp and active mode.

I did some testing of erlang, ejabberd and megaphone.  I have to say, the results were not what I was hoping for.  

On its own, erlang consumes around 2kB of memory per process and 10kB of memory per TCP connection.  ejabberd uses around 100kB per connection.  This is bad news for megaphone because I was hoping to save a lot of memory by shrinking the number of connections down to one, but if the TCP connection only accounts for 10% of the total memory usage per connection, then megaphone is not going to enable significantly larger numbers of connections per server.

Initial testing with megaphone bears this out...unfortunately.  When using megaphone, ejabberd still uses about 100kB per user.  At 1,000,000 users, while megaphone might save 10GB, if the server still needs roughly 90GB, it's not going to make much difference.  

Another issue is that, during testing, ejabberd issued a number of 404 result codes in addition to 200's.  The headers returned appeared to be the same as for 200 messages, but the message body was empty in the case of a 404 --- as would be expected.  Currently, the megaphone protocol does not return a status code --- if things are to move forward, then this will have to be changed.

Next time: what to do.

gen_tcp and active mode

Previously...

• I was able to send and receive messages.
• I changed the pass-through from single to multi-threaded.
• IT ACTUALLY WORKED!

One of the things I came across while developing megaphone and working with erlang has been the notion of "active mode" when dealing with TCP sockets.  When using gen_tcp, you can open a socket in active mode or passive mode.  

In active mode, when a packet of data arrives at the socket, the process associated with the socket gets an erlang message.  In passive mode, the process must ask for data via gen_tcp:recv.  

In a language like C or Java one would always use what is more or less passive mode: you perform a read to get the next block of data.  Coming from that background I had trouble understanding what the rationale behind active mode was.  I think now I understand a little better.

Languages like C and Java do not have the notion of message passing that erlang does, hence active mode does not make as much sense.  With erlang, I can use active mode and create one process (thread) that handles both sending and receiving data:

loop(Socket) ->

    receive

        { tcp, ReceiveSocket, Data } -> do something...

        { write, Data } ->

            gen_tcp:send(Socket, Data)

    end,

    loop(Socket).

Note that the "write" message would have to be sent from another process.

I find this a more natural way of handling socket I/O than the C/Java approaches that I've used.  

Next time: some results from testing with ejabberd.

Communicating

Previously...

• I changed the format exchanged by ECM and megaphone.
• I was able to send and receive messages.
• I changed the pass-through from single to multi-threaded.

I found and resolved the problem that was preventing the system from working: in this case it was prepending the last character from the previous message to the front of the next message.  After that change everything magically worked.

IT ACTUALLY WORKED!

I was able to send messages without waiting for an incoming message, I could connect several times through the same port.  It was beautiful.

The actual code for doing this is depressingly small.  I attribute this to

1. My incredible programming skill
2. (mostly) The task wasn't that complex

A lot of time got used up trying to understand what was going on in ejabberd and becoming more familiar with erlang and nodejs.

At any rate...

IT ACTUALLY WORKED!

(muhahahahaha!)

Next time: some interesting observations on active vs. passive TCP modes in erlang.

One Stop Processes

Previously...

• I got Pidgin to work with the pass-through.
• I changed the format exchanged by ECM and megaphone.
• I was able to send and receive messages.

In my last installment I mentioned a problem where I was unable to send messages from the client unless it was receiving a message from someone else.  It turns out that the problem related the inherent nature of the old version of the code.

The pass-through only had a single process (erlang's version of a thread) to send and receive messages.  What's more, a client sends a POST to the server and then when the server has some data, the server responds with the content of the data in the body of the response.  That's not quite how it works, but close enough.

The problem was that, while the single thread was blocking, waiting for data from the server, new data from the client, for example when a client tries to send a message, has to wait until either the server has something for the client or a timeout takes place.

The multi-client version of the code deals with this issue by having separate threads to send and receive data over the TCP connection.  After a bit of head-banging I figured out the problem.

In a continuation of the whole deja-vu thing, I ran into the problem where the process was not the owner of the socket and was therefore getting socket closed errors.  I still don't understand why I was getting this because I was using code along the lines of:

PID = spawn (some module, some function, some args),

ok = gen_tcp:controlling_process(Socket, PID)

That should avoid the whole mess, but I got the error anyways.  I replaced this with:

gen_tcp:controlling_process(Socket, self())

I ran this from inside the process instead of from the process that spawned it, that seemed to clear things up.

Next time: more results from the new, multi-threaded pass through

Over 10 Messages Delivered

Previously...

• I made progress with a simple BOSH pass-through.
• I got Pidgin to work with the pass-through.
• I changed the format exchanged by ECM and megaphone

After a bit of work, I am able to connect to ejabberd using pidgin and megaphone.  To avoid some problems I am having with some plugins that I use with pidgin, I switched over to JWChat, a javascript client for BOSH systems.

JWChat is an awesome BOSH client, what's more it's quite fast when coupled with ejabberd.  Shout out to Stefan Strigler --- you did a great job with the development of this thing.

I am able to send and receive messages now, but there are some quirks.  While I can receive messages to a client connected to megaphone, messages sent from such a client seem to "stick" and avoid being delivered until the next message for the client is received.  I thought this might be caused by using the default value for TCP's "no delay" option (default value is false), but the problem persists when that option is set to true.  

An additional annoyance is that the socket that listens for new megaphone connections goes into a "time_wait" state if I restart ejabberd, necessitating a 60sec or so pause between cycles.  I tried using gen_tcp:close but that does not seem to help.

Next time: (hopefully) progress on the send/receive message front.

This Really Should be Easier

Previously...

• I got a simple pass-through to work.
• I made progress with a simple BOSH pass-through.
• I got Pidgin to work with the pass-through.

I decided to try and extend the pass-through to be more complex, rather than trying to make the multi-client version work more like the single client version.  This was partially because the multi-client version currently uses raw TCP instead of making use of the node HTTP server, but also because I like the idea of having something that works that I can come back to at each step.

At this point, I'm trying to modify the ejabberd module and the nodejs program to use connection IDs and content lengths.  One additional change that I made was to do away with fixed fixed headers in favor of a variable length header.  

The new header looks like this:

    <connection ID>|<content length>|<content>

The basic differences from the old style headers is that 1) the connection ID and length fields are not zero-padded, and 2) the length field is now the content length field: it contains the character count of the content field instead of the length of the entire message.

Next time: connection IDs and content length fields.

Simple ejabberd Module

Previously...

• I decided to try a simple pass-through.
• I got a simple pass-through to work.
• I made progress with a simple BOSH pass-through.

After a bit more work, I got the BOSH pass-through to work.  Here it is:

-module(mod_simple).
-export([start/2, stop/1, start_module/2, data_loop/1]).
-behavior(gen_mod).


-include("ejabberd.hrl").


start(Host, [Port]) ->
    spawn(?MODULE, start_module, [Host, Port]).


stop(_Host) -> ok.


start_module(Host, Port) ->
    {ok, Socket} = gen_tcp:listen(Port, [binary, {active, false}]),
    ?DEBUG("~p is now listening on ~p:~p", [?MODULE, Host, Port]),
    module_loop(Socket).


module_loop(Socket) ->
    {ok, Client} = gen_tcp:accept(Socket),
    ?DEBUG("Got connection", []),
    PID = spawn(?MODULE, data_loop, [Client]),
    gen_tcp:controlling_process(Client, PID),
    module_loop(Socket).


data_loop(Socket) ->
    case gen_tcp:recv(Socket, 0) of
        { error, closed } ->
            ?DEBUG("connection closed", []);
            
        { ok, Data } ->
            StrData = binary_to_list(Data),
            ?DEBUG("received ~p", [StrData]),
            { _ResponseCode, _Headers, Response} =  
                ejabberd_http_bind:process_request(Data, {{127, 0, 0, 1}, 1234}),
            ?DEBUG("response ~p", [Response]),
            gen_tcp:send(Socket, Response),
            data_loop(Socket)
    end.

The module receives control from ejabberd via the start function.  It immediately calls start_module to fire up something that will hang around and listen for connections.  start_module starts up a gen_tcp listener then goes into module_loop to accept individual connections.


The module is intended to work with a nodejs part that connects to it.  In this, the single-connection version, the nodejs part serves no useful purpose but I am including it because the multi-connection part will need it.

Getting back to the erlang code, when a new connection comes in, the module_loop function starts up a new process (data_loop) and then goes back to waiting for more connections.

data_loop expects to receive the body of a BOSH message which it passes on to ejabberd_http_bind process_request.  For whatever reason, process_request wants an IP address along with the data, so I give it 127.0.0.1:1234  process_request responds with a response code, a set of HTTP headers and the body of the response.  data_loop ignores everything but the body of the response, which it forwards on to the nodejs program.

Not sure whether or not the response code needs to be returned or if the response headers are needed.  For now the values are discarded, but in the future I may modify the megaphone "protocol" to include them.

Here is the nodejs program:


var util = require('util'),
    net = require('net'),
    config = require('./config.js').data,
    http = require('http');


var clientPort = 8280;
var ejabberdPort = 7280;
var response = null;


var serverSocket = net.createConnection(ejabberdPort, "localhost");
console.log("connected to server");


serverSocket.on("data", function(data) {
    if (response != null)
    {
        var headers = {
            'content-type' : 'text/xml; charset=utf-8',
            'content-length' : data.length
        };


        response.writeHead(200, headers);
        response.end(data);
    }
}); 


var server = http.createServer(function (req, resp) {
    req.on("data", function (data) {
        serverSocket.write(data);
    });
    
    response = resp;
});


server.listen(clientPort);


console.log("listening on port " + clientPort);

The program is much simpler than the multi-connection version because, well, it only has to deal with a single connection.  I like not having to deal with headers and CRLF sequences, so I may end up using the httpserver approach with the multi-connection version; especially since the multi-connection version does not work at this point.

Next time: trying to apply these results to the multi-connection version.

Another Step Forward

Previously...

• I noted that the megaphone replies were missing the content-length header.
• I decided to try a simple pass-through.
• I got a simple pass-through to work.

I worked on getting a reasonably simple, single-connection program to work.  It's not quite there yet, but it's a lot further along than the multi-connection program.  


One thing that I did differently was that the ejabberd side was much simpler.  All that is really needed for an ejabberd module is to create a start function and a stop function.  To use the module create a line in the ejabberd.cfg file in the section for modules.  For example:


{modules,
    {mod_adhoc, []},
    ...
    {mod_simple, [7280]}
}


In this example, a module called mod_simple gets passed the parameter 7280.  The function would look something like this:


-module(mod_simple).
-export([start/2, stop/1, start_module/1]).
-behavior(gen_mod).
-include("ejabberd.hrl").


start(_Host, [Port]) ->
    spawn(?MODULE, start_module, [Port]),
    ok.


stop(_Ignored) ->
    ok.


start_module(Port) ->
    <do whatever>.


In the example above, the value passed to start for Port would be 7280.  


Next time: more ejabberd module details.

A Modest Success

Previously...

• I noted that ECM is a piece of junk.
• I noted that the megaphone replies were missing the content-length header.
• I decided to try a simple pass-through.

I tried the simple pass through and it worked!  This, however, is not quite the victory one might hope for.  The real purpose of doing something like that was to ensure that BOSH through node was working at all --- the fact that it did work means that there is at least a chance of megaphone working.

The next step is to set up an extremely simple pass through that goes into ejabberd instead of using the regular BOSH interface via port 5280.  If I can get that to work, it will show not only that my code really stinks, but also that my goal is very achieveable...even in the real world!

Next time: the results of this revised, evil master plan.

Another Perfectly Good Theory

Previously...

• I solved the endless loop problem.
• I noted that ECM is a piece of junk.
• I noted that the megaphone replies were missing the content-length header.

After last time I had resolved to add a content-length header to the messages in the (as it now turns out) vain hope that it would allow pidgin to get a bit farther.  This hope was vain.  It did not come to pass.  Hence it was a vain hope.  

This is different from a vein hope in that it had nothing to do with blood flow.  If it were vein, then perhaps I could start a blog about vampires and angst; which, truth to be told, would not make it much different from this blog.  But I digress.  Also, I am trying to fill out a modest blog posting.

At this point I need inspiration.  As in other times when I am at this point, I turn to beer.  

One of the things that beer has given me is the idea of trying to create a very simple pass-thru that, very simply, just passes the data from the pidgin side to the ejabberd side.  I figure that if I can make it there I can make it anywhere.  It's up to you, New York, New York!

But I digress again.

Next time: the results of the pass-thru.

Content Length Required

Previously...

• I solved the undelivered data problem and added heartbeats.
• I solved the endless loop problem.
• I noted that ECM is a piece of junk.

In an attempt to get Pidgin and megaphone to talk to each other I tried looking at a conversation via wireshark.  Pidgin was using port 5280 (BOSH) in this example, thereby bypassing megaphone.  Here is some of the conversation:

POST /http-bind/ HTTP/1.1

Host: ubuntu2

User-Agent: Pidgin 2.6.6 (libpurple 2.6.6)

Content-Encoding: text/xml; charset=utf-8

Content-Length: 225

<body content='text/xml; charset=utf-8' secure='true' to='ubuntu2' xml:lang='en' xmpp:version='1.0' ver='1.6' xmlns:xmpp='urn:xmpp:xbosh' rid='2652978132620311' wait='60' hold='1' xmlns='http://jabber.org/protocol/httpbind'/>HTTP/1.1 200 OK

Content-Length: 764

Content-Type: text/xml; charset=utf-8

Access-Control-Allow-Origin: *

Access-Control-Allow-Headers: Content-Type

<body xmlns='http://jabber.org/protocol/httpbind' sid='860957cf51a1a6420b82b482853285a914afba43' wait='60' requests='2' inactivity='30' maxpause='120' polling='2' ver='1.8' from='ubuntu2' secure='true' authid='219217664' xmlns:xmpp='urn:xmpp:xbosh' xmlns:stream='http://etherx.jabber.org/streams' xmpp:version='1.0'><stream:features xmlns:stream='http://etherx.jabber.org/streams'><mechanisms xmlns='urn:ietf:params:xml:ns:xmpp-sasl'><mechanism>PLAIN</mechanism><mechanism>DIGEST-MD5</mechanism><mechanism>SCRAM-SHA-1</mechanism></mechanisms><c xmlns='http://jabber.org/protocol/caps' hash='sha-1' node='http://www.process-one.net/en/ejabberd/' ver='yy7di5kE0syuCXOQTXNBTclpNTo='/><register xmlns='http://jabber.org/features/iq-register'/></stream:features></body>

Consider now the conversation that I see going on between Pidgin and ECM:

HTTP/1.1 200 OK
Content-Type: text/xml; charset=utf-8
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: Content-Type


<body xmlns='http://jabber.org/protocol/httpbind' sid='48f09e6c35ad89c27f0d82ae348110f58137d251' wait='60' requests='2' inactivity='30' maxpause='120' polling='2' ver='1.8' from='ubuntu2' secure='true' authid='539841753' xmlns:xmpp='urn:xmpp:xbosh' xmlns:stream='http://etherx.jabber.org/streams' xmpp:version='1.0'><stream:features xmlns:stream='http://etherx.jabber.org/streams'><mechanisms xmlns='urn:ietf:params:xml:ns:xmpp-sasl'><mechanism>PLAIN</mechanism><mechanism>DIGEST-MD5</mechanism><mechanism>SCRAM-SHA-1</mechanism></mechanisms><c xmlns='http://jabber.org/protocol/caps' hash='sha-1' node='http://www.process-one.net/en/ejabberd/' ver='yy7di5kE0syuCXOQTXNBTclpNTo='/><register xmlns='http://jabber.org/features/iq-register'/></stream:features></body>

In addition to being gibberish, it also shows that the responses from megaphone are not including content-length headers in their responses.  This is likely (one of) the problem(s) that is causing the two of them to fail to communicate.  

So solving this problem will cause everything to just magically work.

I'm serious this time.

Next time: a solution to this problem.

ECM Stinks

Previously...

• I noted that data was not getting delivered to ECM.
• I solved the undelivered data problem and added heartbeats.
• I solved the endless loop problem.

For some time now I have been getting the following error when I try to close a connection via pidgin:

/home/cnh/megaphone/ecm/Exchange.js:84

me.ecm.unregisterExchange(me);

         ^

TypeError: Object [object Object] has no method 'unregisterExchange'

    at Socket.<anonymous> (/home/cnh/megaphone/ecm/Exchange.js:84:10)

    at Socket.emit (events.js:64:17)

    at TCP.onread (net.js:348:51)

I finally resolved to fix this problem only to discover a host of other problems.  This has led me to the inescapable conclusion that ECM really, really stinks.  

This is not, perhaps, so much of a surprise as I threw the thing together with minimal understanding of how nodejs works, but I had hoped that the basics were there.  Unfortunately I was wrong.

The good news is that, while it basically does not work, it is fairly easy to understand.  The next order of business is to basically go back to the drawing board with the thing and try to cobble together something that has a chance of working.

End of the Endless Loop

Previously...

• I modified megaphone to send packets.
• I noted that data was not getting delivered to ECM.
• I solved the undelivered data problem and added heartbeats.

It turned out the endless loop on the ECM side was a relatively simple problem.  Here is the method that was receiving the packets:

    me.processDataFromServer = function (data)

    {

console.log("processDataFromServer: " + data);

        //

        // check for a complete packet before adding additional data.  This is not 

        // supposed to happen, but check anyway

        //

        if (me.packet && me.packet.complete)

        {

            me.processPacket();

        }

        if (null == me.packet)

        {

            me.packet = new Packet(data);

        }

        else

        {

            me.packet.add(data);

        }

        while (me.packet && me.packet.complete)

        {

            var leftovers = me.packet.leftovers;

            me.processPacket();

            if (leftovers)

            {

                me.packet = new Packet(leftovers);

            }

        }

    };

The problem was that processPacket was supposed to set "me.packet" to null, but in the case of a heartbeat packet, it was not doing this.  Once that was resolved, the endless loop well...ended.  

At this point, it seems like data is being received by ECM, but Pidgin never sends a reply to the reply from ejabberd.

For next time: is ECM really receiving a reply?

node and erlang Heartbeats

Previously...

• I got megaphone to receive packets.
• I modified megaphone to send packets.
• I noted that data was not getting delivered to ECM.

Last time, I resolved to create a "heartbeat" style message between megaphone and ECM.  In the course of doing this I noted that, if I tried to connect ECM to megaphone after a crash of ECM, data would not get delivered.  That is, if I start up ejabberd/megaphone and then connect ECM to it, data gets delivered to ECM.  If I try reconnecting ECM to megaphone, things don't work.

The nodejs side of the heartbeat does most of the work.  When ECM first connects to megaphone, it sets up an interval timer.  Each time the interval timer fires, ECM sends another heartbeat.  The systems distinguish heartbeats from regular messages by using connection ID 0 for heartbeat messages.  

Here is the (modified) code for ECM that starts up the heartbeat process:

        me.socket.connect(config.server.port, config.server.host, function () {

            console.log ("now connected to " + config.server.host + ":" + config.server.port);

            setInterval(me.sendHeartBeat, 5000);

        });

This fragment only includes the code for the connect.  The function to send the heartbeat is also pretty simple:

    me.sendHeartBeat = function()

    {

        var hbmsg =

            "0000000041|"

            + "00000000000000000000|"

            + "heartbeat";

        me.socket.write(hbmsg);

    };

The code for receiving a message had to be modified to check for heartbeat messages:

   me.processPacket = function ()

    {

        console.log("packet: " + me.packet);

        if (me.packet && me.packet.complete)

        {

            if (me.packet.connectionId == 0)

            {

                console.log("heartbeat");

            }

            else

            {

                var connectionId = me.packet.connectionId;

                var intCID = parseInt(connectionId, 10);

                var exchange = me.sockets.get(intCID);

console.log("got exchange: " + exchange + " about to send reply");

                exchange.sendReply(me.packet.content);

                me.packet = null;

            }

        }

    }

The megaphone side is simpler than the ECM side.  It just looks for messages with a connection ID of 0 and immediately sends a response for them:

process_request(ConnectionID, Data) ->

    if

        ConnectionID == 0 ->

            Response = "heartbeat",

            ?MODULE:send(ConnectionID, Response);

        true ->

            NewData = parse_packet(Data),

            ResponseData = ejabberd_http_bind:process_request(NewData, {{127,0,0,1}, 1234}),

            Response = response_data_to_response(ResponseData),

            ?MODULE:send(ConnectionID, Response)

    end.

So now everything is peachy.  Except that the ECM side simply prints out "packet: [object]" and "heartbeat" over and over again.  Next time I will try to figure out why this is.

Totally Confused

Previously...

• I changed the system to stop sending "raw" HTTP.
• I got megaphone to receive packets.
• I modified megaphone to send packets.

I thought there was a relatively straight-forward problem with ECM - specifically, it looked liked it was getting the equivalent of a null pointer exception.  So I looked into the problem, fixed what seemed to be wrong and tried again.

This, of course, did not work.

So far as I can tell, megaphone is formatting the data correctly, it's sending the data, it just doesn't seem to be getting to ECM.  At least all the time.

Sometimes, the packet arrives at ECM, I get a message and whatnot, but most of the time ECM just sort of sits there with an expectant look.

It's times like these that make me glad I've been giving shout outs to various countries, because this means I can forget my troubles by going out and drinking the celebrated alcoholic beverage of said country.  I have to admit, however, that I'm partial to wine and beer --- I tend to avoid distilled stuff.

The current problem could make me change my ways in that vodka is looking awful good when I contemplate my troubles.  

One thing I'm thinking of doing to test out the whole connection between ECM and megaphone is to create a "background" process for megaphone that periodically sends data to ECM.  This would take the form of an "I'm alive!" style message that ECM would lovingly receive...and then promptly throw away.  Nevertheless, it would give me a warm fuzzy that something was actually happening in terms of the connection between megaphone and ECM.

Next time: implementing this marvelous plan.

HTTP Responses

Previously...

• I turned it on and...it didn't work.
• I changed the system to stop sending "raw" HTTP.
• I got megaphone to receive packets.

When we last visited our intrepid developer, I was looking at some sort of error where megaphone was complaining about the response it was getting from ejabberd_http_bind.  The response seems to be of the form:

{ <code>, <list of HTTP headers>, <data>} 

whereas megaphone wants to see something akin to just 

<data>

Judging by how ejabberd talks to pidgin over port 5280, it looks like the response should be something like this:

HTTP/1.1 <code> <description>

<header>: <value>

<header2>: <value2>

...

<header-n>: <value-n>

<blank line>

<data>

Where every line should be terminated with a carriage-return, linefeed combo.

I wrote some code to format the header list as a list that is closer to what is needed:

format_headers(Prefix, []) ->

    Prefix;

format_headers(Prefix, [{HeaderName, HeaderValue} | Rest]) ->

    NewPrefix = Prefix ++ io_lib:format("~s: ~s\r\n", [HeaderName, HeaderValue]),

    format_headers(NewPrefix, Rest).

And then something to format the response into a string:

response_data_to_response ({ResponseCode, Headers, Data}) ->

    HeaderList = format_headers([], Headers),

    HeaderStr = lists:flatten(HeaderList),

    ResponseCodeStr = response_code_to_string(ResponseCode),

    ResultList = io_lib:format("HTTP/1.1 ~w ~s\r\n~s\r\n~s", [ResponseCode, ResponseCodeStr, HeaderStr, Data]),

    lists:flatten(ResultList).

A very important note: you must use "lists:flatten" on the output of io_lib:format, otherwise you will get a listinstead of a string.  

At this point, I am bailing on the return code: I just print "OK" rather than interpreting the code correctly and printing out the correct response code.  Hence the definition for response_code_to_string:

response_code_to_string(_Code) ->

    "OK".

This appears to have a vague chance of working, except that, when the response is being sent back, ECM chooses that point to crap out:

/home/cnh/megaphone/ecm/ECM.js:149

var exchange = me.exchanges.get(intCID);

                               ^

TypeError: Cannot call method 'get' of undefined

    at [object Object].processPacket (/home/cnh/megaphone/ecm/ECM.js:149:32)

    at Socket.<anonymous> (/home/cnh/megaphone/ecm/ECM.js:187:7)

    at Socket.emit (events.js:67:17)

    at TCP.onread (net.js:327:14)

For next time: what is going on with ECM?

P.S. Another shout out to Russia, since that fine country's people have seen fit to actually look at this blog :-)