Neverwinter Nights 2 Blog

Currently loading of resources n NWN2 is a big mess

 

The following all applies to atleast all of:

TLK files these are specially bad today as they are loaded wrong by the toolset

GUI files

Haks

2das

special effects

journal

scripts

models

blueprints

sound files

terrains

worldmap files

graphics files

 

The way I would hope it worked: (in each case the directory in program files having lower priority than the corresponding directory in my documents)

Game built in resources would have lowest priority

then the corresponding game directory: hak,tlk, sounds etc.

Then campaign resources: Basically we could put in any of the types of things into the campaing directories

then the in module resources: Again Basically we could put in any of the types of things into the module or module directories.

then override: Again Basically we could put in any of the types of things into the override directories.

 

 

But I would not make haks automatically override anything unless they are part of module properties or in the actual campaign/module/override directories.

 

 

Also on the general topic:

* The lack of an official hak editor. -> somethign the communtiy can make, so not primary priority

* The lack of an official UTF-8 string editor. -> somethign the communtiy can make, so not primary priority

* The lack of a place where "stand-alone" (unaffiliated with modules and campaigns) haks can live (override and data folders not really being a good home for such things). -> override should work well

* The lack of documentation on the order in which things are loaded and from where (with order being very important, as it establishes what files have "last say"). -> Very true

* Data files can be read in from about eight different places in myriad formats. ->very true, but some of the files cannot be read form some of the directories

* .haks must be explicitly called out from individual campaigns and modules. ->as it should be

* There is no logging of .hak and override content being loaded to see gets read in last and from where. ->true

* There is no data screen where campaigns, .haks, and override files can be seen, much less managed. ->true

 

But add to that:

Loading of content in multiplayer:

-Not loaded before char generation

-No indication what resources can be serverside and what have to be in clients and in what directories

-No usefull feedback to player when some resource is missing

-Some resources not working in some of the places

-No ability to check consistency of any resource so it has not been tampered by the player.

I agree totally with weby on the above mentioned items. I especially agree with not having haks autoload based on what directory they're in. The suggestion above seems to move away from the organization of separate folders for content and move toward communal areas based on module/campaign where we may have to keep 3 copies of certain files if they're used in three different ways. I may have read it wrong too. Either way, I would rather the way haks and other resources are organized and called be kept so that the module controls this from it's properties. If making an external control would cause these to load better then maybe having a module/campaign specific .inf file created from the module settings that would be used to do the loading as you suggest above. Maybe even make the default paths for haks, etc. something configurable inside the module that would write those paths to a .inf file. Either way I do agree with weby that the priority and loading of resources needs to be controlled by the module/builder and not automatic.

 

It is nice to see you guys tossing these ideas out for comment though. Thanks for soliciting our input!

Personally, I would like for all of the resource specific to a particular module or campaign to be stored in its own folder under the campaign's or module's parent folder in a structure similar to this:

 

\module/campaign's directory in My Documents

|-\hak

|-\modules (for locally stored modules, not needed for PWs or hosted games)

|-\override (for testing or hot fixes)

|-\pwc (for PWs or hosted games, not required for locally stored modules)

|-\tlk

 

This should also have priority over any of the hak/data files stored anywhere else and should be automatically loaded when the player starts the module or connects to a multiplayer hosted module. Having all of the resources stored in a single place like this makes it easier to deploy a module on a client's machine. Also, why have a different directory for campaigns and stand alone modules? Couldn't it just use one location for both? In the case of campaigns, they would have multiple modules in the modules folder or for stand alone modules, only 1. For example:

 

Player A connects/loads Module A. As soon as the module starts to load, it then checks to see if there are any files stored in the \hak, \tlk, \pwc, \override folders on the client machine and loads them into memory. At some point, either before or after character generation, the 2da files loaded into the client's memory are verified against those stored on the server to ensure the player didn't modify them. Then the player is allowed to connect. In the case of a non-multiplayer module, this wouldn't need to occur since the player is free to load the module in the toolset and modify it anyway. If any of the haks, 2das, tlks, or other module specific files don't match the MD5 hash of those on the server, then the connection should be refused with a message informing the player to download the latest content from the modules website (or nwvault ) Then they simply have to grab one file and extract it into the modules folder under the Neverwinter Nights 2 folder in My Documents and then attempt reload/reconnect.

 

That being said, there should be the option of the module builder to use haks created by other custom content teams, such as the PRC and the CEP. So in the module options there should be a check box (unchecked by default) that allows the module to load unaffiliated haks from the \hak folder in the Neverwinter Nights 2 folder in My Documents. Only the haks listed in the module/campaign properties should be loaded and not a blanket loading of all haks stored in that location. Explicit loading of haks only is the best route to go.

 

The only parts of the actual module file(s) that should be exported for client side use is the walkmesh information and visual/sound effects. Everything else should be stored on the server and processed there. This would greatly assist in preventing characters from modifying in module resources when they are connected. After those, the largest concern is 2da files changes. These should be checked when the player connects and each time the player levels up or perhaps area transitions. Either that or have the client lock the files on the process level (like MS Word locks document files) to prevent changes to the 2das while the client is loaded. Just tossing out ideas there, nothing really solid, as I am sure you can tell.

I think all the suggestions here are excellent - there is far too little information and knowledge about what gets loaded, where its loaded from, etc. As it is, every piece of custom content you see on the Vault usually includes the instructions 'dump it into your override folder'

 

Then, when different people download mods, 1/2 of them dont work right, as they have things in their override folders that are breaking functionality.

 

The worst culprit is the toolset itself - very often, when you open a script to examine it (say, the nw_c2_default9) from the OnSpawn handler in a creatures properties (or any item, or anything else that has handlers), then close the script in the Script editor, a ZER0 kb file (thats right - completely empty) gets created in your override directory. So all of a sudden, key functionality is gone - creatures dont perceive you, nothing works, etc. Has happened to me many times.

 

As the override folders are so full of everyone's custom content, its sometimes hard to spot this behavior. Having a completely clean override would certainly help.

A few issues to bring to attention:

 

1. GUI XML files and Hakpacks not co-operating

 

They do not work together. I tried in my beta 1.05 and I put a context menu XML file in the override folder. This worked fine. I put one in a hakpack, associated it with my 1 area test module, it didn't even load the module! (created the hakpack using this tool: http://nwvault.ign.com/View.php?view=NWN2T...Detail&id=9 )

 

These are the 3 files I used to test: http://finaldeath.co.uk/nwn/files/nwn2/GUI_problems.zip

 

---- End Server Options ----
[Sat Mar 31 01:11:50] Player  [] () Joined as Server Admin 1
[Sat Mar 31 01:11:50] Player  () Joined as Player 1
[Sat Mar 31 01:11:50] Loading Module: test_hakpack_gui
  TRANS: [Sat Mar 31 01:11:50]Not decompressing as not loading from a previously visited module.
  TRANS: [Sat Mar 31 01:11:50]Setting up Stall Event now.
Couldn't load the Hak Pak File "test_contextmenu_hak.hak"
Could not load the Module.
Missing required HAK file.

 

In addition, TLK files didn't load from hakpacks either.

 

Music, I suspect, also is the same.

 

These 3 types of file could be consolidated, so people who create custom content can distribute it more easily, and module builders can extract it to campaign directories if they wish, or use the hakpacks as is.

 

Just in case why you might wonder why it is more useful, apart from choice, hakpacks provide:

• A contained set of files with easy ways to instruct module users to use. Module builders don't have to worry about file conflicts ("what overrides what? what zip do I do first? what do I merge?!?! there is no manual to this!)
• Allows hierarchies of hakpack files to be used like all other file types can be in
• Allows, with hierarchies, proper "addons" or "optional" hakpacks, ie; you include a placeholder GUI file for an addon feature, which simply doesn't do anything, and a second hakpack holds the actual addon feature which overrides the placeholder in the correct hierarchy, but there are many other examples of this being needed.

These don't work in a hakpack file:

• GUI .ini files (not needed in 1.05 I think right?)
• GUI XML files.
• Music (I think, I have not got time to test just now)
• TLK files - vastly important

I forget if multiple talk table files can be used, or if they need to be merged, but simply having them able to be put in a single hakpack file anyway (or set of haks, which all go in the same folder, the hak folder) would be a good idea.

 

In addition, campaign folders suffer from not having access to, I think, TLK and music files. (please correct me if I am wrong).

 

Inconsistency as to where things go make creating anything for the game very hard. Its hard enough having 2 folders for game data - one in the documents one and one in the game folder, and not knowing the hierarchy between those!

 

2. Logs

 

There are no logs of what is loaded (at least, what hakfiles are loaded and which locations, and in what order). Feedback is needed to find conflicts.

 

Or they are all over the place.

 

EVERYWHERE, like, one in EVERY place, and no one knows where they all are. local temp folder (which is never deleted either, but thats another several hundred megabyte story), my documents, the game install folder, anywhere I missed?

 

Can't there be one log file location for all of these? all the crash reports and server logs? all the error logs? Perhaps an option in the config files :D

 

It also hung when there was an invalid / not found hak. The game should at least go into console more / report an error / go back to main menu right?

 

It would have to be optional, and always off by default, but if a debug setting for finding out:

• What archives / locations are loaded
• In what order
• What separate files load
• With conflicts reported

Would be a good idea. Having each one optional (ie; conflicts separate from what archives are loaded) would be good too.

 

Not being able to find bugs is confusing at the moment with no feedback - my issue above with putting an xml file in which crashed the loading and no feedback is part of this.

 

3. GUI script exploits

 

Since GUI scripts can execute server-side scripts in multiplayer, and GUI scripts can be edited by players (and there is no option or way clients can readily check for "just using the defaults" or whatnot), there is a "security" risk, ala cheating, available to some players.

 

It is good, in its way, that only gui_ scripts can be executed. It is bad that there are dozens in a default game install. If a server admin does not know to do so, the default death resurrection script could be abused. There are others too, beyond the basic henchmen behaviour changes (which are fine, except they can mess up other peoples henchmen) there are the item naming script and a few others that could be abused in this way.

 

In future expansions I fully expect more default ones will be added (and thus must be, else they'd have to be created by a builder to work at all!). There either should be some documentation or more checks put in such scripts, else it is easier then combing server code to exploit the game!

 

If I am wrong on any of these points please say, I would like to be proved wrong really.

Jasperre: kivi found that the module will load after another (recognized) file, such as a 2da, is added to the hak. Once this is done, the custom contextmenu.xml is not recognized.

 

Which ties in well with my thoughts on custom content:

While specific uses may vary, there needs to be consistency. Every single resource type should load uniformly whether contained in a Hak, Override folder, or a Campaign folder. This includes gui xml, tga, dds, mdb, wav, ncs, nss, ut*, sef, pfx, bbx, and all the other myriad file types that NWN recognizes as resources.

 

 

A few issues to bring to attention:

 

1. GUI XML files and Hakpacks not co-operating

 

They do not work together. I tried in my beta 1.05 and I put a context menu XML file in the override folder. This worked fine. I put one in a hakpack, associated it with my 1 area test module, it didn't even load the module! (created the hakpack using this tool: http://nwvault.ign.com/View.php?view=NWN2T...Detail&id=9 )

 

These are the 3 files I used to test: http://finaldeath.co.uk/nwn/files/nwn2/GUI_problems.zip

 

---- End Server Options ----
[Sat Mar 31 01:11:50] Player  [] () Joined as Server Admin 1
[Sat Mar 31 01:11:50] Player  () Joined as Player 1
[Sat Mar 31 01:11:50] Loading Module: test_hakpack_gui
  TRANS: [Sat Mar 31 01:11:50]Not decompressing as not loading from a previously visited module.
  TRANS: [Sat Mar 31 01:11:50]Setting up Stall Event now.
Couldn't load the Hak Pak File "test_contextmenu_hak.hak"
Could not load the Module.
Missing required HAK file.

 

I just tested this and it works fine. I tried a single hak with only contextmenu.xml in it -- I associated it to a module and the module functions perfectly well (loads, runs, the stuff is there). Where did you place your hak file? The hak needs to go to the My Documents path and the hak folder there. This has worked fine since I tried back at least as far as 1.04 and it still does work in beta 1.05.

I'm all for organization. really, still I would like to see the tiles.2da and tileset.2da files be able to be read from within a hak file...you're gonna need that anyway (among other things) before any kind of organization system will be of any use.

 

With CC tilesets not really being supported by the game (by "not really" I mean from within a hakpak and not requiring use of the override) its kinda cramping Hellfire and my (Robinson Workshop) style

I'm am very happy to see this disaster of data organization in NWN2 being addressed. Fortunately, the solution should be easier to implement than what is already there.

 

For .mods. Really, they should contain all the localized resources used by the module and SPECIFIED by the module designer, IN ONE CONVENIENT PLACE <----most important thing. A .mod should be an archive of files and directories to keep things separate from other .mods and for easy transport and management. Thus, EVERYTHING is managed through the .mod. It is the ROOT of all resources, if you will. This makes the most sense.

 

Please, lets not have a .mod and THEN have directories for haks and tlks OUTSIDE the .mod that the .mod depends on. This is where things are now and it its too confusing. Sharing common haks between mods is far far less of a concern than compact, self-contained management because disk space is far more abundant and the server can be made to not redundantly load resources (by using their crc hash or unique id).

 

The simplest solution is always the best. Put everything into the .mod and for that matter most of the bigger PW's and modules will use directory mode anyway. All their custom content, regular content, and myriad resources can then be managed under one tree. This allows for convenient use of other tools like version control FOR ALL RESOURCES.

 

A simple CRC checksum of sorts between the server and the client can detect any tampering. Easy.

 

The override folder stays as is.

 

I've abandoned my NWN2 endeavors for the past 5 months because of this headache. So hopefully it can be fixed.

 

Sessh

Actually, i kindly disagree. I have never encountered any naming conflict in hak ressources, and i personally wouldnt want to handle 6gb files. our pw is already at 2gb, the haks already at 1 gb.

 

modules shuld be allowed to have everything in them, i think there everyone can agree. hak files are a way to have additional files, again .. everything. so, if you dont want hak files, just dont use one. and if you want to put everything in your mod file, do so.

 

the easiest solution i can think of would be:

 

.cam files -> allowed to have everything in them, for campaigns

.mod files -> allowed to have everything in them for a module

.hak files -> allowed to have everything in them for a module / campaign that _adresses_ that hak (you dont have t, your choice)

.pwc files -> allowed to have everything in them for a specidifc PW

.override files -> allowed to have everything in them, but it would be tidier if in the main override folder only .overrides would be allowed to exist ... you may also use subfolders with loose files

 

override of course changes to the top if the files in it are not rule relevant (maybe it woulrd be a great idea to include a module switch for this)

 

---

 

so, if somebody fears that name confusionand versioning conflicts might interfere with his module ... well, simple, assign no haks and put all the stuff in your module.

 

if somebody does NOT fear that, as I certainly dont, dont force them to have files those big.

 

Sessh btw forgets that PWs might work in directory mode, but they still have to transfer pwc files including all the hak stuff then.

 

---

 

this would be the best solution i can think of. tidy, and everybody has control of whats loading for his module. one may distribute the whole CEP with his module, but one doesnt have to.

 

and, of course, as everything is .hakable, .modable, .overrideable .... there is absolutely no need anymore for music, ui, tlk folders (tlk -> part of module or hak, just as you desire)

Hi,

 

I am only just starting to use NWN2 for modding and found the structure of the files etc most confusing, so I am glad this is being looked at.

 

I don't know if it is practical or not, but even removing the entire list of (duplicated) directories in the MyDocuments would make a start. I.E Just have one main diretory (the NWN2 directory) as a place to store all relevant files, be they therefafter in a "default" subdirectory folder for the official campaign stuff or "custom" for anything added by a modder.

 

E.g. At the moment, I don't know what the difference is between placing UI scripts in my MyDocuments/ui folder or the NWN2/UI folder. (Apart from it does not work one way?)

 

Why have both folders then? (Especially if one cannot be added to for any reason.) It confuses which one of the two to use.

 

Personally, I don't like placing stuff in MyDocuments anyway, especially when I want to work on modding.

 

Lance.

Actually, it is quite simple ... you should never have to place anything in the install directory. Everything is under my documents.

 

Reason: This is Vista-standard. On many systems, users may not be able to touch all the other folders at all.