Question About Game Assets & Data

[Luridis]

I have a curiosity that you guys might be able to answer. I've mentioned before that I do occasional desktop development for work, so I'm familiar with many of the tools available there. Also, I see games like Neverwinter Nights, and many others of course, with huge client side data stores. These are usually in flat files and/or some sort of compression + indexing cabinet. Hell, some MMOs can have folders spanning gigabytes of information.

 

My question is... with the proliferation of desktop SQL engines, why don't game developers use these? I could understand when they needed to run as a server type service. But some of the more recent compact engines are little more than a lib that does the reading of the database file. Seems to me that organizing and especially patching client data could be so much easier in that scenario. Of course I'm not a game developer so you probably know something I don't, and that's what I'm trying to understand.

 

Tanx!

[Walsingham]

If it's not cheeky would you mind elaborating a bit on what is meant by each system? I like to understand these things, but most online resources assume too much background information.

Edited May 21, 2013 by Walsingham

[Blarghagh]

I don't know anything about engines, but I found this using google:

 

 

 

 

An SQL database is not nearly fast enough to use for realtime reading and writing game information. Such data is almost always kept in memory, in traditional data structures.

There may be some benefit to using an embedded database such as SQLite for certain types of data, eg. static data that doesn't change during gameplay but does change during development. This could then be deployed as part of the final game where SQLite is only really used when loading up the game for the first time, or when starting a new level, etc.

However there are many downsides too - it is hard to patch individual parts of the data when they're stored in a single database file, it is not ideal for many types of complex data that games need (and which you said you'd store outside - but will have references to and from things inside), it is not very flexible when you need to change the schema, it is not necessarily backwards compatible after you change the schema, etc.

For these reasons, most game developers will just use their own format. Professional developers who are performance conscious sometimes go one step further and save the in-memory data structure directly to disk so that it can be loaded in with a minimum of processing.

And if you really need text-based tabular data that is easily edited, you could use a simple text based format, such as CSV, XML, JSON, YAML, etc.

 

And

 

 

A database isn't fast, using a database is a whole lot slower than traditional memory access. The reason is simple, a database is dynamic, there is a bunch of overhead attached to it, the queries need to be parsed, hashes need to be computed and other stuff.

There are is only one advantage from using a database and that's persistence. You can run one or multiple applications with one database over a long period of time without having to fear data lass. But game clients have absolutely no need for that.

So, you want to get every object which is less than 1000px away?
That would be:

List<Entity> retVal;
for(entity in entities)
if(Distance(entity.pos(), to) < 1000)
retVal.append(entity);
return retVal;

Now what would you think would a database do if you asked it to get every object less than 1000px away?
It would do exactly the same! Just with a bunch of overhead which would make this one extremely simple operation cost about 100 times the processor time (depending on how many entities you have). Databases ain't no magic.

Don't use databases for anything else but server applications.

 

Sorry about that quote tunnel, the Bb Code ****ed up.

Edited May 22, 2013 by TrueNeutral

[Luridis]

Ah, thanks TrueNeutral...

 

I see... However, I do think databases would be fast enough if they were in an enterprise configuration. Or possibly on computers with SSD drives. The reason I would think so is because business databases are quite fast and grab huge chunks of binary data in short order. But, we're talking SMP Oracle back-end stuff and not MySQL mini on someone's laptop in power-saving configuration.

 

Walsingham:

 

Here are a couple of links describing each of the storage methods I was talking about...

 

Game file: http://nwn.wikia.com/wiki/.2da

Game file: http://en.wikipedia.org/wiki/MPQ

 

On the other hand, SQL is a language for accessing Database Management Systems. DBMS is how a lot of, if not most of, the large complex data files are stored and structured today. For example: Each post on this forum isn't stored as a text file somewhere, it's likely written in a database and stored on a server in one large file. When you go to a page the server executes a number of queries that collects the information for the page you want to see.

Edited May 22, 2013 by Luridis

[Walsingham]

Thanks for the responses. No time to read today, but will have a look over the weekend.

[Orogun01]

It's definitively a venue worth exploring, however I think that with current gen complexity an SQL database will be too slow to run at realtime. It definitely sounds better than cloud gaming.

[alanschu]

 

 

it's likely written in a database and stored on a server in one large file

 

Remember when The Witcher patches were several gigabytes in size (effectively redownloading the whole game), because changing one bit of data in the file required a redownload of the entire file?  (Granted, patching can be done better on Steam, but that's neither here nor there... it illustrates the advantages of simplicity, however).

 

 

Other things to note is that "fast" can mean two different things.  High bandwidth is still fast, and you won't care if your multi-terabyte query takes 10 seconds to complete, because to you you think "wow, I got all this information in 10 seconds!"  If all your game processes are stalled out for 10 seconds though, because it's waiting for that information, it'd be unplayable (this is an extreme example, to help illustrate the point).  With games, however, you're often dealing with milliseconds (at best) for everything, and that latency is very, very important.  If you're attempting a large number of queries (especially table modifications), you'll start to hammer on your DBMS performance too.

 

SQL (and other) databases have been used in games (usually MMOs, whether casual or hardcore), however (I know we used them in development for Dragon Age Origins 1 and 2, and the Frostbite editor interacts with a database repository as well).  Databases are more useful when working with vast amounts of data, which most video games that aren't "massively multiplayer" just don't have.

 

Here's a write up that has some GDC influences within the contexts of

http://doublebuffered.com/2006/10/30/why-sql-sucks-for-mmorpgs/

 

And a brief abstract citing challenges from SIGMOD 2009 (this will probably be most beneficial for you to read).

http://www.cs.cornell.edu/~sowell/2009-SIGMOD-Games-Tutorial.pdf

[anubite]

I'm not sure why you would even need a database, even for a very complex RPG. For MMOs - duh - you're storing instance data, player account data, world data et cetera, but for a single-player RPG you just have a couple megabytes worth of data on what's going on where in a dungeon and then you have a file that saves the state of areas you're not in, which won't be queried until you enter a new area, which prompts loading.

 

You'll need to be more specific, honestly. What are you envisioning? You can't use SQL databases for rendering anything, as it would be way too slow (you use lists, arrays and such). What do you need to search for in a game that requires a look up? If everything is indexed, you'll get it fast. If you're not indexing, chances are, it's because you don't have a lot of data.

 

I mean, you don't really need to sort anything if your list contains <50 items. When are you going to have more than 50 items in a single player RPG? Even if you have 1000 items, you can do one of the cheaper sorting algoritims and be fine. I only see the point in a complex database once you move beyond that, to 50k or 100k items you need to search through for whatever reason, which will never happen in a single player game - unless it's Dwarf Fortress-esque complexity.

[Luridis]

I'm not sure why you would even need a database, even for a very complex RPG. For MMOs - duh - you're storing instance data, player account data, world data et cetera, but for a single-player RPG you just have a couple megabytes worth of data on what's going on where in a dungeon and then you have a file that saves the state of areas you're not in, which won't be queried until you enter a new area, which prompts loading.

 

You'll need to be more specific, honestly. What are you envisioning? You can't use SQL databases for rendering anything, as it would be way too slow (you use lists, arrays and such). What do you need to search for in a game that requires a look up? If everything is indexed, you'll get it fast. If you're not indexing, chances are, it's because you don't have a lot of data.

 

I mean, you don't really need to sort anything if your list contains <50 items. When are you going to have more than 50 items in a single player RPG? Even if you have 1000 items, you can do one of the cheaper sorting algoritims and be fine. I only see the point in a complex database once you move beyond that, to 50k or 100k items you need to search through for whatever reason, which will never happen in a single player game - unless it's Dwarf Fortress-esque complexity.

 

I don't know if you're playing "Adventure" on the Atari 2600 or what... Couple of megabytes of game data? 50 items? Perhaps you should go right click your NWN2 game folder and select "properties." I don't currently have it installed, but I'll venture the size is greater than 1GB and tens of thousands of files.

 

I wasn't just talking about a save... I'm talking scripts, dialogues, tiles, textures, meshes, binary data... You name it.

 

I understand it a little more lately as I'm experimenting with systems programming. I had no idea it was this way, but most of the IO libraries out there only offer sequential reads as a stream. That I find odd, considering that even the Commodore 64 provided code for random access to files via side index in the 1541's tiny 32KB controller ROM. How much I am seeing in modern operating systems, even within the Win32 API, that is based on *nix architecture is mind-boggling.

 

Anyway, sequential reads being the only method of access, it makes more sense that game data built in numerous files of widely varied size would perform better when written as a zipped chunk. Not only would it reduce the fragmentation created by so many individual files, but also be easier to read with a stream cursor.