Fixing some broken JSON in the bundles?

[ydaraishy]

Some of the JSON in the stock gamedatabundles seem to be invalid, e.g., Brigandine_Armor_Exceptional in items.gamedatabundle has an EquippableComponent with ItemModsIds has '"ItemModsIDs":[,"2f868d39-3962-4a62-a656-041121931423"]'. Perhaps all EquippableComponents have this problem?

 

This chokes on automated JSON processing Is this fixed in 1.1, or can it be fixed for a future patch?

 

(This can be worked around with stream editing, but "[," looks like an invalid JSON sequence anyway...)

Edited June 6, 2018 by ydaraishy

[Kohwalter]

Still has this comma in all ItemModsIDs, although they have no impact whatsoever in the game as far as I know nad no problem when editting the json files with notepad++.

[ydaraishy]

I don't use notepad++, I use jq, which presumably is a bit stricter about correct JSON.

[BMac]

I took care of this for the next patch (1.1.1, I believe).

[peardox]

I'm parsing the gamedatabundle files in PHP

 

In v1.1.0 there are six badly formatted files in gamedata...

 

// $file = 'exported/design/gamedata/ai.gamedatabundle'; // CTL

// $file = 'exported/design/gamedata/audio.gamedatabundle'; // CTL

// $file = 'exported/design/gamedata/global.gamedatabundle'; // CTL

// $file = 'exported/design/gamedata/items.gamedatabundle'; // Syntax

// $file = 'exported/design/gamedata/progressiontables.gamedatabundle'; // CTL

// $file = 'exported/design/gamedata/statuseffects.gamedatabundle'; // Syntax

 

1) At the end of a line CTL means there's a return char inside a double quoted string e.g. in audio.gamedatabundle there's a line that looks like this

 

"AnimationEvent": "1h_punch

"

There's an 0x0A after 1h_punch, I believe this MAY occur at the start of a string as well (not going thru them all again)

 

2) At the end of a line Syntax means it's a '[. "Key": "Value"' as explained by @ydaraishy

 

One last concern is that progressiontables.gamedatabundle has some keys padded by spaces (it's the only one) the first occurrence being...

 

{

"Note": "Fire Breath ",

" Category ": " General ",

" UnlockStyle ": " AutoGrant ",

" ActivationObject ": " Self ",

" AddAbilityID ": " c7c80cae - a692 - 4237 - b266 - 536801b18e5b ",

" RemoveAbilityID ": " 00000000 - 0000 - 0000 - 0000 - 000000000000 ",

" Prerequisites ": {

" MinimumCharacterLevel ": 1,

" PowerLevelRequirement ": {

" Class ": " None ",

" MinimumPowerLevel ": 0

},

" RequiresAbilityID ": " 00000000 - 0000 - 0000 - 0000 - 000000000000 ",

" Conditional ": {

" Operator ": 0,

" Components ": []

},

" VisibilityConditional ": {

" Operator ": 0,

" Components ": []

},

" IsMutuallyExclusiveUpgrade ": " true "

}

}

Edited June 11, 2018 by peardox

[nstgc]

I just fix them manually with Vim before beautifying them with Python.

Edited June 11, 2018 by nstgc

[peardox]

The point is that the exports should be "proper" JSON

 

Personally, Notepad++ gives me very quick fixes now I know what's wrong.

 

I've got a PHP script that identifies the bad ones on my Linux server which I fix via a share

 

The Syntax ones are the worst.

 

When I started debugging exported I had to do manual binary discovery (cut half the possible offending lines out til you get something that works then work on half the ones I cut)

 

VERY time consuming...

 

Now I'm aware of the issues the next update will be easier to process

[nstgc]

The point is that the exports should be "proper" JSON

 

Personally, Notepad++ gives me very quick fixes now I know what's wrong.

 

I've got a PHP script that identifies the bad ones on my Linux server which I fix via a share

 

The Syntax ones are the worst.

 

When I started debugging exported I had to do manual binary discovery (cut half the possible offending lines out til you get something that works then work on half the ones I cut)

 

VERY time consuming...

 

Now I'm aware of the issues the next update will be easier to process

Which ones were those? I've only needed to hunt down miss place commands and white space. That's less than a minute with Vim without any special scripts. However, I've only looked at those in gamedata/.

[peardox]

I can only speak about the things I know for fact (unlike the other thread) so here I'll lay it out for you.

 

I'm building a MySQL DB via reverse engineering everything in exports with the aim to make a guidebook app for web / mobiles.

 

This results in me being REQUIRED to understand the structure of every file, I don't find some random string that appears to work, I have to prove it to myself first (hard work)

 

Much of what I'm doing has a specific aim but I often come across things I park in the back of my brain until required which why I was able to answer you in 10 mins (OK - it was a fail but I had the knowledge)

 

I've got a copy of Pillars on my Linux box (just the exports dir) to automatically process everything and so I can fix the screwed up JSON

 

Did you know that you can link directly from a QuestGiver to the Quest? That's my next goal (that one's actually fairly easy)

 

I've got to re-writ the monster file tomoz though - stupid social media

 

"Wahh - it's not hard enough, Simon, can you make it harder for me?"

 

Sheesh (I wanna do creatures anyway)

 

If you can do MySQL I'll happily share clean data

[ydaraishy]

Again, using a JSON processor like jq can give you byte offsets where there are issues and can output prettyprinted JSON from the bundles. You can patch these errors with 

sed -e 's/\[,,/\[/g' | tr -d '\n'

in any case.

[kilay]

wrong post, delete it

Edited June 17, 2018 by kilay

[peardox]

I took care of this for the next patch (1.1.1, I believe).

 

Nope

 

I made a copy of exported on my local Linux box.

 

I wrote a PHP script to check out the JSON 1,147 files in exported/data skipping exported/design/fogofwar/* as they're some format I can't identify.

 

There are all easily and quickly fixable with notepad++ using search & replace for the SYNTAX ones, for the CTRL ones simply make everything one line in notepad++ and they're fixed.

 

Analysis

 

All files marked JSON_ERROR_CTRL_CHAR have line breaks in the middle of a value

 

exported-1.1.1/design/chatter/chatter.chatterbundle

JSON_ERROR_SYNTAX - Ends in },]} should be  }]}

 

exported-1.1.1/design/gamedata/ai.gamedatabundle (BOM)

JSON_ERROR_CTRL_CHAR

 

exported-1.1.1/design/gamedata/audio.gamedatabundle (BOM)

JSON_ERROR_CTRL_CHAR

 

exported-1.1.1/design/gamedata/global.gamedatabundle (BOM)

JSON_ERROR_CTRL_CHAR

 

exported-1.1.1/design/gamedata/items.gamedatabundle (BOM)

JSON_ERROR_SYNTAX - Multiple cases of [, should be [

 

exported-1.1.1/design/gamedata/progressiontables.gamedatabundle (BOM)

JSON_ERROR_CTRL_CHAR

 

exported-1.1.1/design/gamedata/statuseffects.gamedatabundle (BOM)

JSON_ERROR_SYNTAX - Multiple cases of [,, should be [

[kilay]

 

I took care of this for the next patch (1.1.1, I believe).

 

Nope

 

I made a copy of exported on my local Linux box.

 

I wrote a PHP script to check out the JSON 1,147 files in exported/data skipping exported/design/fogofwar/* as they're some format I can't identify.

 

There are all easily and quickly fixable with notepad++ using search & replace for the SYNTAX ones, for the CTRL ones simply make everything one line in notepad++ and they're fixed.

 

Analysis

 

All files marked JSON_ERROR_CTRL_CHAR have line breaks in the middle of a value

 

exported-1.1.1/design/chatter/chatter.chatterbundle

JSON_ERROR_SYNTAX - Ends in },]} should be  }]}

 

exported-1.1.1/design/gamedata/ai.gamedatabundle (BOM)

JSON_ERROR_CTRL_CHAR

 

exported-1.1.1/design/gamedata/audio.gamedatabundle (BOM)

JSON_ERROR_CTRL_CHAR

 

exported-1.1.1/design/gamedata/global.gamedatabundle (BOM)

JSON_ERROR_CTRL_CHAR

 

exported-1.1.1/design/gamedata/items.gamedatabundle (BOM)

JSON_ERROR_SYNTAX - Multiple cases of [, should be [

 

exported-1.1.1/design/gamedata/progressiontables.gamedatabundle (BOM)

JSON_ERROR_CTRL_CHAR

 

exported-1.1.1/design/gamedata/statuseffects.gamedatabundle (BOM)

JSON_ERROR_SYNTAX - Multiple cases of [,, should be [

 

 

Thnks for the effort

Could you share the fixed files? just to save time

Edited June 24, 2018 by kilay

[peardox]

Sure

 

I've branched them to exported/json skipping fogofwag (which don't compress well)

 

I've applied json_pretty_print to them all and tagged a .json extension to them so e.g. items.gamedatabundle -> items.gamadatabundle.json

 

Basically they're easy to read now - some files were already legible but others had no formatting

 

They're about 5.5M if I RAR them, 10M for ZIP or TGZ

 

They expand to 319M

 

You can have the json ones or just the fixed un-processed gamedatabundle etc set

 

Gimme a little while to set up a subdomain on my server to house them all and I'll bung them up

[peardox]

OK - one quick shoddy website later here are all the repaired files

 

https://poe2.peardox.com/exported/

 

design.* are the original files while json.* are prettified version with '.json' added to the filename

[kilay]

OK - one quick shoddy website later here are all the repaired files

 

https://poe2.peardox.com/exported/

 

design.* are the original files while json.* are prettified version with '.json' added to the filename

 

Great, neverending thnks