A new way of thinking about database storage and web apps

If you have been developing web apps 'for years', then you are probably used to thinking about database storage in terms of SQL.

When I first looked at Redis for example, my first impression was simply .. WTH ?  Why so bare bones ?

My second thought was .. "Just look at the performance and scalability of this stuff -  so it might be worth going old-skool on the database side of things just for the performance boost."

Yes - there is the performance and scalability angle, but that as it turns out is missing the point entirely.

One of the more difficult things to get your head around with this "new" way of working is getting over the fact that all of the sudden, the system requirements for database storage might be a LOT simpler than you are used to thinking.

In my existing system design (from the PHP/MySQL interpretation of the system), there is a LOT of rather complex data modelling, and all of that complex data is stored in MySQL.  So of course, I started this project off by looking at remodelling all of that as a huge collection of loosely defined Redis types.

.... Duh ... WRONG !

Looking at the data a second time, it is apparent that a lot of it ... (well, most of it actually), is there to help restore state for each and every  new incoming web request.

When you move from a stateless paradigm (short lived PHP code being invoked from apache via an incoming request) to a long running stateful node.js applet  .... you can just as well maintain state in memory / program variables without even having to store any of that in any sort of database.

All that is needed is to work out which sections of that mass of information in the program variables need to be persisted at all ... and simply model those parts using 1 of the 5 nifty data structures provided by Redis.

As it turns out, the bulk of the complex data exists in the Order of Battle data structures. For any given battle, these ORBATs remain constant. All you need to track (and persist) is a little bit of data attached to each unit, which provides the current casualties / fatigue / morale state / orders queue. 

That is only half a dozen state variables for each unit in the game. No big deal. No complex relations need to be defined in the database either - just a bunch of flat variables that are related to each unit by name.

Those things turn out to map very nicely to a couple of simple Hashes and Lists in Redis.  

All that is needed then is some way of representing the ORBATs for the battle in progress.  How to edit these / store them / load them ?

At first thoughts about this problem, I was thinking that maybe MongoDB might be the way to go for this, since the ORBATs are a tree structure with variable fields in each set of data, so that maps well to a MongoDB document.

In the existing model, I have the ORBATs being created and edited in XML format (using TreeLine), and then a bunch of PHP code to load these XML files, parse them, and shove them into MySQL tables.

If I could hack that to load them into a MongoDB document instead, the problem is half fixed,

...  2nd big DUH of the day !!!

Got to remember that we are not in PHP land anymore, we are in Javascript world.

And that means .... JSON. 

No need for XML anymore, as JSON is human readable, so its easy enough to knock up ORBATs as straight JSON format files.  And that means no need for a special editing program - just edit them directly using vi.

Next problem - how to load those JSON files, parse them, and convert them into data structures in the node.js application to initialise the state of the game ?

... 3rd big DUH of the day !!

As of version 0.5, node has a 1 liner to do this :

require("name-of-my-orbat-file.json");

So there we go.

Dont need no ORBAT editor at all  (vi JSON file does the trick)

Dont need no ORBAT parser (JSON is native to node)
 
Dont need no Object Mapper  (The JSON data IS the object)
 
Dont need no backend to front-end translator (Same language running at both front-end and back-end)
 
Dont need no ORBAT database storage  (the ORBAT is constant for each battle)
 
Dont need no complex relations in the persistent storage (1 Hash per unit does the trick)
 
Dont need no cronjobs to clean redundant data (Redis EXPIRE is your friend)

Thats all completely and utterly wunderbar, and something of a revelation. For once, developing web applications doesn't feel like a broken tower of babel. Nice !

What is quite amazing is that just a few lines of Javascript code in a single  file at the backend, and a few simple JSON format text files, and a few simple Redis variables  .. gives a complete running web application.

There is a whole mountain of infrastructure that you simply DONT NEED ! Bundling this thing up for distribution suddenly becomes a whole lot simpler with that in mind.

Getting your head around that, I think, is the key to making some progress with these  tools.

The next part of the app deals with synching all of this state information across multiple front end clients, and multiple back end servers (eg - AIs and other cool things) .. and doing it all in real time with low latency and low traffic over the wire.

Thanks to SocketStream, and Redis pub/sub .. there is equal simplicity in the solution to that potential can of worms as well.

So I guess the moral of the story so far ... is that if you are coming into this development environment from a more traditional web application development background ... the quickest way forward might be to forget everything you know, and give yourself the opportunity to approach your own system design problem from a very simple and very clean slate.

The solutions are all there, sitting right under your nose ... but they might be much more trivial and elegant than you are expecting. That takes a little getting used to.

Looking forward to the next phase of the development, as this is starting to make more sense now. Its a fun environment to work in too.