A new way of thinking about database storage and web apps

If you have been developing web apps 'for years', then you are probably used to thinking about database storage in terms of SQL.

When I first looked at Redis for example, my first impression was simply .. WTH ?  Why so bare bones ?

My second thought was .. "Just look at the performance and scalability of this stuff -  so it might be worth going old-skool on the database side of things just for the performance boost."

Yes - there is the performance and scalability angle, but that as it turns out is missing the point entirely.

One of the more difficult things to get your head around with this "new" way of working is getting over the fact that all of the sudden, the system requirements for database storage might be a LOT simpler than you are used to thinking.

Project 1

Project 1

Project1 is A NodeJS application to manage Napoleonic miniature wargames.

Really friggen huge miniature wargames with thousands of miniatures on giant tables and dozens of players.

Using a computer backend for managing the game allows for 4 significant improvements in game play, that are simply impossible using traditional wargaming systems :

1. No need to abstract out detail to speed the game up. Bookkeeping becomes a non-issue.
2. Replace the abstraction of 'Turns and 'Phases' in the game with a smooth flow of real-time events and player interactions.
3. Real fog-of-war, using off-table campaign maps and hidden movement.
4. Remote game participation over the internet.

To do this properly is a real technical challenge. One of the most difficult things I have ever worked on I reckon.

Existing traditional web frameworks .. unfortunately, don't exactly cut the mustard either. The existing HTTP model of stateless requests presents some problems to the flow of information as well. It can be done, but it requires a huge tower of babel of code, and some ugly practices over the wire (long polling, timers, etc)

The large and complex data models needed to describe all the unit states and their hierarchy is a poor fit for the RDBMS  approach. Scaling a traditional PHP/MySQL app to work smoothly needs a lot of ugly hacks to fit the game.

So it was time to get back to the drawing board, and write something from scratch in C, using a persistent in-memory data store of the whole game at the back end, and communicate via socket based messages to native apps on the front end.  That's a big project, similar in scale to JANUS .. which is used by the Defence Dept to model C3 aspects in professional kriegspeil.

And then along comes ... NodeJS, Redis data store, and SocketStream.

.. just as cheap Android tablets become widely available.

hmmm.  All this stuff is an absolute perfect fit for the problem domain. Perfect !  I would even suggest that this combination of tech is designed from the ground up to manage those sorts of massively detailed multi-player tabletop miniatures battles that for decades we have only been able to dream about.

... and to do it all with a real time event queue rather than clunky abstracted 'turns', or clunky abstracted 'units' is a total revolution.

Embedded arduino sensors on the tables, RFID tags on the units, and handheld Android tablets for each player = a monstrous, ridiculously detailed warfare simulation for the OCD perfectionist crowd.

Thanks to the new programming paradigms presented by these tools in combination, the solution to the problem is now relatively easy, and relatively cheap to pull off. Excellent !

GitHub repo to be posted soon.