Chanter Rime and Frost traps get saved, causing mobileobjects.save to grow out of control

[Depili]

I think this deserves it's own post instead of being buried on page 5 of the savegame issue thread. My post there: http://forums.obsidian.net/topic/72764-quick-saving-and-loading-times-increased-greatly-after-20-hours/?p=1615960

 

So in short the Chanter "Rime and Frost" chant creates traps when it is triggered and thoise traps get saved into mobileobjects.save file causing it to grow out of control. This in turn causes saves to take way longer than they should. Oddly the chant creating fire traps doesn't seem to be affected with this bug.

 

Here are the statistics in a nutshell:

$ du -h MobileObjects.save
44M     MobileObjects.save
$ strings MobileObjects.save | grep Rime_and | wc -l
3232
$ strings MobileObjects.save | grep Root | wc -l
3308

That is 97% of all store objects!

 

Explanation for the commands: First one shows the size of the MobileObjects.save, secong extracts all strings (text fragments) from that binary file, selects all that match the "Rime_And" stringfragment and counts them, giving us the count of 3232 Rime and Frost traps stored, last one does the same for substring Root that seems the be the start of new saved object data, giving the total count of 3308 saved objects.

 

Workaround would be to avoid the rime and frost chant like plague, a fix would be to write a script to strip the saved traps from saves and to prevent new ones from getting saved at all.

 

Edit: Apparently my first grepping miss-counted the amount of Rime and Frost traps, as the string "Rime_and" appears twice per trap, so a better match is:

 

$ strings MobileObjects.save | grep "Rime_and_Frost_Trap.prefab" | wc -l
1614

Which leaves 50% of objects being traps. But the rime and frost trap definitions seem to be some of the longest objects.

Edited March 31, 2015 by Depili

[Depili]

Digging deeper to the strings in MobileObjects.save. Here are all the unique strings sorted by number of occurances in my save: https://gist.github.com/depili/1117c9f30047c4985a14

 

From this its quite easy to see that there are 1725 objects of the same unknown type, 1614 rime and frost traps, and that Obsidian loves cloning:

437 Player(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)(Clone)

Maybe a bit excessive, don't you think? =) (The string "(Clone)" appears total of 53k times in MobileObjects.save).

Edited March 31, 2015 by Depili

[Yonjuro]

....

$ strings MobileObjects.save | grep "Rime_and_Frost_Trap.prefab" | wc -l
1614

Which leaves 50% of objects being traps. But the rime and frost trap definitions seem to be some of the longest objects.

 

 Thanks for looking into this. It might be worth sending a PM to Roby Atadero (the Obsidian Dev. who responded to the other thread) if you don't get a response. I imagine the team is busy and could miss this if it falls too far down the page.

[Depili]

 

....

$ strings MobileObjects.save | grep "Rime_and_Frost_Trap.prefab" | wc -l
1614

Which leaves 50% of objects being traps. But the rime and frost trap definitions seem to be some of the longest objects.

 

 Thanks for looking into this. It might be worth sending a PM to Roby Atadero (the Obsidian Dev. who responded to the other thread) if you don't get a response. I imagine the team is busy and could miss this if it falls too far down the page.

 

 

Send a PM. Hope the eventual fix also includes scrubbing of existing savegames, as otherwise it won't be of much use for people already suffering from this.

[Depili]

Looking into the amount of data tied into the Rime and Frost traps, one trap (everything between two Root strings) is about 26kb, meaning that the 1600ish traps in my 44Mb MobileObjects.save take up about 42Mb of it. (This is all just from looking at one random trap).

[eRe4s3r]

This is a very nice find. I mean.. the finding of it is nice, the thing you found isn't so nice

 

26kb for a single trap definition in a save is also very very... crazy.  I guess the compression was added because otherwise the save would have violated steams cloud save guidelines. So you should be very careful if ye plan make to a mod that changes compression. It can blow peoples cloud save storage up if they forget to disable it and there is no easy way to delete savegames from there.

 

Either way, that's a very interesting find

[Depili]

Each saved trap seems to contain a serialization of the chanter that cast the spell and array of (probably all) "teams" on in the game, and they take up tons of space. The teams are there probably to set who can trigger the trap (monsters don't trigger traps on the dungeon floor, only the ones you place yourself).

 

One puzzling association are the "player (Clone)(Clone)...." strings, there are few with various amount of the (Clone) tag appended to them, in the worst case the engine is creating new cloned entities of the chanter for the traps, which might also increase the memory usage while playing if so.

[Bubbles]

Very helpful indeed, thank you!

[Depili]

Looking at the difference between two MobileObjects.save files from two different saves suggests that there might be some evil lengt/checksum fields in the file, which might make a quick-and-dirty savegame scrubber impossible. Tempted to still have a go at it since I haven't done any tampering of unknown binary blobs in the past

 

Current plan is to just first split the stored objects (Making some assumptions on their boundaries per the start and end of the file) and then toss out all chunks containing the Rime and Frost string, reassemble the file and hope for the best...

[Depili]

First experiment was a success, Deleted a around 20Mb chunk from the MobileObjects.save file to see if it causes save/load errors, but everything seems promising, as the game loaded just fine and arena transitions are now way faster!

 

I cut the chunk starting after the terminating null byte of a string previous to the "Root" string starting a trap chunk and ended at a randomly chosen null byte terminating a chunk. So there appears to be no checksum or a global length info for the file anywhere.

 

Next I need to do a cleaner solution by writing some ruby and toss all the traps away, and only the traps. Some additional challenge: There is a chunk with the party data, that includes the chanter and will thus quite easily match the trap-filter, so need to avoid that.

 

Edit: Something unexpected apparently happened also, after loading the edited game, doing a area transition and saving again the MobileObjects.save is now down to 2,5Mb, so something might have triggered some garbage collection as a side-effect. If so I will leave my experimentation at this and just be happy.

 

Edit2: Well, it wasn't such a success, that brute force test apparently also got rid of 4 recruited NPCs, so need to really write that script

Edited March 31, 2015 by Depili

[eRe4s3r]

Maybe if some Obsidian dev *hint* comments on this and mentions that a fix is already in the next patch you can save yourself the hassle? (Just thinking loud) what you have ahead of you is definitely not easy.. trap definition searches I tried have tons of mismatches with other parts in the file...

 

Also there is still the matter as to *why* loading a 40mb file takes forever. That should not be the case, read speed on an SSD can handle that within a second and 3000 objects to serialize aren't exactly "a lot" for a modern engine. So something else (aside from the trap bloat) must still be weird in the engine itself too. Something that also makes area loads (without saving) longer than it feels they should be.

 

Anyway, I have not a clue about save and load systems in games. But I know Unity engine 4 has *tons* of issues with asset (especially texture) loading so.. I hope Obsidian can find the causes for both the trap/save bloat as well as increasing LOAD times for areas. There is no reason why areas take longer and longer to load, even on a savefile that is not affected by massive bloat.

Edited March 31, 2015 by eRe4s3r

[Sleazebag]

Wow, no wonder saving takes so long in my game, I've been using that chant all the time after getting the stronghold. Does it affect zone loading times too? 

[Roby Atadero]

Got them PM, thanks for the heads up. We'll start looking into this.

[Depili]

Wow, no wonder saving takes so long in my game, I've been using that chant all the time after getting the stronghold. Does it affect zone loading times too? 

Well, game creates a autosave on each zone load....

[Depili]

Got them PM, thanks for the heads up. We'll start looking into this.

 

\o/

 

Wrote a quick and dirty ruby script that is going towards the right direction: https://gist.github.com/depili/20afa3ca0ccabd427d62 but while it successfully strips 40M worth of traps out of the MobileObjects.save I still have something messed up, as while loading the modified save is successful I'm missing much of my equipment and some partymembers. Will do another try with slightly altered splitting if that ends up working.

 

If interested this shows the distribution of the traps in my save: https://gist.githubusercontent.com/depili/7d32a41573ca94f6ef15/raw/24d25d1eb1b9c0a93967d0ebb79c8ecf53d0ef00/gistfile1.txt each dot is a non-trap chunk, each X is a trap.

 

Edit: Actually after minor modification the script seems to now work, made a bad guess at the chunk boundary, even when the real place was quite obvious in the end. Now my modified savegame loads fine with all party members and equipment and saving times are way down. Use at your own risk etc etc (that is a ruby scrip for unix systems, usage: ./trap_cleaner.rb MobileObjects.save You need to extract that file from your save and then repackage the resultin MobileObjects.save.cleaned file into its place )

 

The whole exercise from finding the bloat to removing it has been quite nice little project for this evening, haven't worked with unknown binary blobs before

 

For others maybe wanting to implement this in their own way this is pretty much what I learned:

The "chunks" in the file start four bytes before the string Root, that is the place to split, then you can toss all chunks containing the string "Rime_and_Frost_Trap.prefab" and keep the rest. Since the traps are mingled around with "good" chunks you really can't do this by hand.

Edited March 31, 2015 by Depili

[99_]

I made an account because I'm having the same bug as described in this thread...

[at0mic]

nice script. ill try it as an IT guy. VMware can easily make a Linux install if u need one. now if only someone could make a script that cleans up all my immortal summoned drakes. lol

[Alamar]

nice script. ill try it as an IT guy. VMware can easily make a Linux install if u need one. now if only someone could make a script that cleans up all my immortal summoned drakes. lol

You don't need linux for this. There is ruby for windows - http://rubyinstaller.org/

[at0mic]

i tried the script along with the no autosave mod and im down to an acceptable-for-now 12 second load time.

http://www.nexusmods.com/pillarsofeternity/mods/14/?

 

someone plz figure out how to scrub out all the immortal summons next!

 

-- interestingly, when i put the scrubbed .save file in the game dir its at 12 mb. soon as the game launches and checks the save game files where you get the continue button , it spikes back up to 45ish MB. its pulling save files from steam. how to stop that? i set the steam cloud sync to disable but it still pulls the saves from steam. wow..  i disabled my network card and its still populates the save directory with all the saves.wheres it coming from?

Edited March 31, 2015 by at0mic

[at0mic]

i think its coming from the .zip package in C:\Users\at0mic\AppData\Local\Temp\Obsidian Entertainment\Pillars of Eternity

[jantjeuh]

Great work guys

[EricKei]

This actually explains a l lot -- I did not have the "saves take much longer to save and load" until after I started using that Chant! I've removed it from the rotation now. Thanks!

[at0mic]

C:\Users\at0mic\AppData\LocalLow\Obsidian Entertainment\Pillars of Eternity\CurrentGame

 

also gets a copy

[Depili]

C:\Users\at0mic\AppData\LocalLow\Obsidian Entertainment\Pillars of Eternity\CurrentGame

 

also gets a copy

I didn't touch anything other than one .savegame file in %USERPATH%/saved games/ -directory, but I didn't work with the most recent save. I would suggest saving your game in a unusual location first to make identifying the correct file to work with easier, as the location will be visible in the file name after the GUID.

[duncandun]

So, as of now it seems like I have completely messed up saving in PoE. Originally I edited my quicksave, which in retrospect may have been a bad idea.

For whatever reason now, every save I create is ****ed and either results with the game just returning to the main menu or a black load screen.

This has extended to a brand new character I made to test it as well.

No idea whats going with that, gonna try reinstalling I guess.

for reference old saves (and my backup of the one I edited) load fine. Any save I've made after I edited it however, is broken.

I'm going to delete all the saves I've made afterwards and see if that fixes it.

 

 

 

C:\Users\at0mic\AppData\LocalLow\Obsidian Entertainment\Pillars of Eternity\CurrentGame

 

also gets a copy

I didn't touch anything other than one .savegame file in %USERPATH%/saved games/ -directory, but I didn't work with the most recent save. I would suggest saving your game in a unusual location first to make identifying the correct file to work with easier, as the location will be visible in the file name after the GUID.

 

 

C:\Users\user\Saved Games\Pillars of Eternity is the path to it.

[Depili]

You might also try to just move the temp files to another location and see if that helps (%TEMP%/Obsidian)

 

But yeah, tinker with your saves at your own risk.,,