New backend system

New backend system

Postby adum » Fri Jan 22, 2010 1:09 am

i've completely overhauled the way the backend stores sliced data. my database was growing out of control -- thanks, holland! :D

the new system is 200x more efficient, which will hopefully last for a good while. it seems to be working and the transition was pretty seamless, but if you see any problems with the new system, please post immediately!
User avatar
adum
Site Admin
 
Posts: 25
Joined: Tue Jun 23, 2009 10:55 pm
Location: Menlo Park, CA

Re: New backend system

Postby ES187 » Sat Jan 23, 2010 9:13 pm

I've had some trouble on Friday afternoon (22 Jan between 14:00 and 16:00 CET) with lines which first showed as full lines but then appeared to be broken at some places (and had to be redone). Don't know of this was due of this transitions or just bad luck with GPS & 3G connections in that specific area. (Hadn't any time to do some slicing today, so don't know if these behaviour is persistant).
ES187
 
Posts: 52
Joined: Sat Jan 23, 2010 8:55 pm

Re: New backend system

Postby eelco » Sat Jan 23, 2010 11:32 pm

Performance of the storage is of course important. But what happens if your userbase increases to 3000? Or 30.000? I guess your server will go up in smoke. :)

I was thinking about a way to lower the stress for your server. The area that any inividual will cover with the App runnign is rather limited, compared to all slices that you have to store in your database.
I find it hard to do the calculations as I have no idea how you store the slices exactly, but I guess the following concept would decrease the number of requests on your server dramatically:

Store slices on the iPhone, add a timestamp, and on next startup, only request all slices in the area that were added since the last timestamp.

Why:
- Once created, a slice will never change. It cannot disappear or change shape. ( this is the main argument )
- Players will cover a rather limited area, so the iPhone should be able to store that data easily
- Queries on your server will be limited: select slice from slices where (distance to iphone < XX km) and timestamp > last_update
- As slices are all stored on the iPhone, it should be able to draw all slices in the area fast, in the right colors, so that players will more often use their iPhone to explore the neighbourhood, and not your webserver. I guess that webserver will have performance trouble anytime soon as well.
- Right now, the app shows only slices that are very near. They are drawn very slow, and they are often glitched (I suspet sync problems due to latency / slow connection).

More precise:
- Only store slices on the iPhone in a limited radius around slices owned by that iPhone itself. That will be the main interest of the player anyway.
- Include a limit on slices to be stored on the iPhone. Those who do indeed cover lots of area in many parts of the world can still hammer your server.
Slicer #153
eelco
 
Posts: 19
Joined: Tue Jan 19, 2010 1:28 pm
Location: #153

Re: New backend system

Postby adum » Mon Jan 25, 2010 1:20 am

thanks for the ideas. my limit was more about data size on the db server rather than load. i think we're good for the foreseeable future...

there is some caching locally. i could improve it, but probably no need unless things get way heavier.
User avatar
adum
Site Admin
 
Posts: 25
Joined: Tue Jun 23, 2009 10:55 pm
Location: Menlo Park, CA

Re: New backend system

Postby eelco » Tue Jan 26, 2010 12:12 am

I think load is an issue, at least for the Google Maps mashup. It does not load at all, often. Do you monitor performance at all hours? I guess the load is highest when you are asleep. ;)
Slicer #153
eelco
 
Posts: 19
Joined: Tue Jan 19, 2010 1:28 pm
Location: #153


Return to Earthslicer

Who is online

Users browsing this forum: No registered users and 2 guests

cron