Solved Very large and thus slow save game file

Status
Not open for further replies.
D

Dakkaron

Guest
#1
Whenever the game has to access data from the "disk", it is really slow on my phone (Moto Z Play, Android 7.1.1, SFS 1.35), so I took a look at the save file, and I think I found the reason.

This happens every time I open the ship editor, press "To Launch" or "Continue". Every time I do that, I have to wait about a minute until the game continues.

All the game's save data is saved to a common shared preferences file, which in my case is a whopping 8.5 MB large. This causes performance problems as shared preferences files aren't very slow. The data is saved as JSON which is then HTML urlencoded and then put into XML.

The first change that would improve the performance by a lot is to break this up into multiple shared preferences files, so only the necessary parts need to be loaded at any one time. Namely, that would be:

"Quicksaves"
"To Launch"
"Persistant"
"Build Saves"
"BuildQuicksaves"
"ToLaunch"
"GameQuicksaves"
"PersistantGameSave"

and one file containing all the rest.

First of, what I noticed is that there are three different quicksaves, two different "To Launch"-values, and two "Persistant" values. To me this seems as if some of these are left-overs from older versions of the game. It would be very good to clear those, instead of just leaving them, since they increase the size of the file by a lot and shared preferences files need to be read whole, even if only a single value actually needs to be read.

Now, when looking at the saves themselves, there seems to be an older, more compact format that contained just IDs for every field in the grid, and a newer, more verbose format, that contains each part and its properties. This takes a lot of space though.

As an example, I'll take my miniature moon probe:
Screenshot_20180512-121749.png
URL-encoded, this tiny ship takes up 3223 bytes of storage. This is taken up in "ToLaunch", but if it is saved it's also in "BuildQUicksaves" and for every ship that is in-game, there is also a copy in "PersistantGameSave". So if you have a few a bit more complex ships, this quickly eats a lot of memory.

Now, URL-encoding JSON is very inefficient, since all of JSONs special characters will be replaced by a three-character HTML escape. So without the URL-encoding this only takes 1757 characters. But since you can't control the URL-ncoding (it is done automatically by the shared preferences functions) it would be worth making the structure of the save file as flat as possible (no unnecessary nested JSON objects) and also it would be worth reducing the length of each variable.

Currently, each part is saved like so:

JavaScript:
{
  "partName": "Solar Panel",
  "position": {
    "x": 9.5,
    "y": 31.5
  },
  "orientation": {
    "x": 1,
    "y": 1,
    "z": -270
  }
}
(I formatted this, so it is easier to read, in the save file, there are no whitespaces.)

URL-encoded, this part takes 169 characters.

This could easily be reduced to this:

JavaScript:
{
  "p": "Solar Panel",
  "x": 9.5,
  "y": 31.5,
  "rx": 1,
  "ry": 1,
  "rz": -270
}
URL-encoded, this would now take only 116 characters. If the name of the part is now replaced with an ID, this number would go down to 99 characters. This alone would save a lot of space, but it could be reduced even further if you would switch over to an even simpler format for each part. In URL-encoding, the tilde character ("~") does not need to be encoded, so it only takes one character. So if you do a format like so: "[id]~[x]~[y]~[rx]~[ry]~[rz]", so in this case "11~9.5~31.5~1~1~-270", you would be down to 20 characters for each part, or an 88% reduction in disk space. Applying this to the 8.5 MB save game file, that I have, this would reduce the file size (and thus the load speed) down to 1 MB. If then all of the unnecessary stuff from older versions is removed as well, I am quite sure, this would be reduced by even a lot more.


Let me know if this kind of code-review is interesting to you.
 
T

TtTOtW

Guest
#3
You seem to know a lot that I don't, but Stef probably does. Why don't you email him?
 
D

Dakkaron

Guest
#4
Sorry, I thought this was the official forum for Spaceflight Simulator. Is it?

What's Stef's email address?
 

Gecko Gekkota

Former Empress of the AE
Forum Technician
#5
Sorry, I thought this was the official forum for Spaceflight Simulator. Is it?

What's Stef's email address?
it is, but i don't think that he has time to be on here, MikeWe is your best chance at contacting him, well, along side E-mailing Stef, he does reply
 
D

Dakkaron

Guest
#8
I actually cleared out all the unnecessary old values from my savegame and now I am down to 198kb.
 
D

Dakkaron

Guest
#12
@TtTOtW: Thanks for the address, I emailed him. Looking forward to his reply.

I wish there was a proper bugtracker somewhere... Maybe have a Github-page that doesn't contain any code but is only used for issues.
 
T

TtTOtW

Guest
#13
One of the main purposes of the SFS community sites is exactly that. Continuous feedback for development purposes. @MikeWe is on the SFS dev team also. And he personally oversees this forum. So give him a shout if you need advanced support ok?
 
#16
I wrote to that email address on wednesday, but I haven't received any answers so far.

Thanks for the info about @MikeWe.
We've reworked the entire save system since we started with 1.4 development, saving is fast and doesn't take as much space as before. I've never seen this thread before but it could've saved us even more time ;).
 
Status
Not open for further replies.