New backend system

[adum]

i've completely overhauled the way the backend stores sliced data. my database was growing out of control -- thanks, holland!

the new system is 200x more efficient, which will hopefully last for a good while. it seems to be working and the transition was pretty seamless, but if you see any problems with the new system, please post immediately!

[ES187]

I've had some trouble on Friday afternoon (22 Jan between 14:00 and 16:00 CET) with lines which first showed as full lines but then appeared to be broken at some places (and had to be redone). Don't know of this was due of this transitions or just bad luck with GPS & 3G connections in that specific area. (Hadn't any time to do some slicing today, so don't know if these behaviour is persistant).

[eelco]

Performance of the storage is of course important. But what happens if your userbase increases to 3000? Or 30.000? I guess your server will go up in smoke.

I was thinking about a way to lower the stress for your server. The area that any inividual will cover with the App runnign is rather limited, compared to all slices that you have to store in your database.
I find it hard to do the calculations as I have no idea how you store the slices exactly, but I guess the following concept would decrease the number of requests on your server dramatically:

Store slices on the iPhone, add a timestamp, and on next startup, only request all slices in the area that were added since the last timestamp.

Why:
- Once created, a slice will never change. It cannot disappear or change shape. ( this is the main argument )
- Players will cover a rather limited area, so the iPhone should be able to store that data easily
- Queries on your server will be limited: select slice from slices where (distance to iphone < XX km) and timestamp > last_update
- As slices are all stored on the iPhone, it should be able to draw all slices in the area fast, in the right colors, so that players will more often use their iPhone to explore the neighbourhood, and not your webserver. I guess that webserver will have performance trouble anytime soon as well.
- Right now, the app shows only slices that are very near. They are drawn very slow, and they are often glitched (I suspet sync problems due to latency / slow connection).

More precise:
- Only store slices on the iPhone in a limited radius around slices owned by that iPhone itself. That will be the main interest of the player anyway.
- Include a limit on slices to be stored on the iPhone. Those who do indeed cover lots of area in many parts of the world can still hammer your server.

[adum]

thanks for the ideas. my limit was more about data size on the db server rather than load. i think we're good for the foreseeable future...

there is some caching locally. i could improve it, but probably no need unless things get way heavier.

[eelco]

I think load is an issue, at least for the Google Maps mashup. It does not load at all, often. Do you monitor performance at all hours? I guess the load is highest when you are asleep.