How can they solve the massive texture problems?

[sb5]

After seeing that many flames in the past I've registered to say that big images can be split up into smaller, manageable chunks of texture. As far as tiling goes you could have unique hand drawn areas that can connect to other areas at intersections for dungeons!

[J.E. Sawyer]

I did some quick calculations, based on wild guesses about the size and number of areas, overhead required for animated objects and so on. What I came up with was an estimate of 6-10GB for the backgrounds.

 

For animated objects (characters, monsters and such) you need the usual textures and such, but they would be smaller. Still, overall we might be looking at a total game size of 15GB or so.

 

That's still a reasonable size to fit on a set of DVDs.

 

Our slightly-above-average dungeon levels contain ~1 gig of render data, uncompressed.

[Ieo]

 

I did some quick calculations, based on wild guesses about the size and number of areas, overhead required for animated objects and so on. What I came up with was an estimate of 6-10GB for the backgrounds.

 

For animated objects (characters, monsters and such) you need the usual textures and such, but they would be smaller. Still, overall we might be looking at a total game size of 15GB or so.

 

That's still a reasonable size to fit on a set of DVDs.

 

Our slightly-above-average dungeon levels contain ~1 gig of render data, uncompressed.

 

 

~1 gig uncompressed per level of dungeon, or for the whole dungeon (with an average how many levels)?

[sb5]

You are missing normal, lighting, shadowing and blocking maps in your numbers

[J.E. Sawyer]

 

 

~1 gig uncompressed per level of dungeon, or for the whole dungeon (with an average how many levels)?

 

 

For a single level.  The number of levels in a dungeon isn't necessarily relevant, but the number of levels in the game is.  This is going to be a large game with a lot of maps.

 

That number does include all renders, btw (final, diffuse, specular, etc.).

[Gorgon]

What's the customary thing with downloadable versions. Do they always include textures for resolutions you can't use ?. Seems wasteful. Save the rainforest and the other people sharing my connection.

 

Just an observation.

[Fearabbit]

For a single level.  The number of levels in a dungeon isn't necessarily relevant, but the number of levels in the game is.  This is going to be a large game with a lot of maps.

 

That number does include all renders, btw (final, diffuse, specular, etc.).

 

 

Okay... so how good is compression?

Because to my knowledge, you can compress a texture down to maybe 15% of its uncompressed size, i.e. roughly 7 times smaller. In order to be downloadable the game shouldn't be bigger than 10 GB so that would give us 7 x 10 = 70 maps.

That doesn't sound like much at all, and we didn't even consider that music, sounds and voice acting need a lot of space too...

 

That said my calculation is probably completely wrong.

 

EDIT: Actually it sounds about right, though I guess the game could be bigger than 10 GB. I just hope it won't be too much, with my ****ty Wifi connection it takes ages to download 10 GB.

Edited September 8, 2013 by Fearabbit

[Sezneg]

 

For a single level.  The number of levels in a dungeon isn't necessarily relevant, but the number of levels in the game is.  This is going to be a large game with a lot of maps.

 

That number does include all renders, btw (final, diffuse, specular, etc.).

 

 

Okay... so how good is compression?

Because to my knowledge, you can compress a texture down to maybe 15% of its uncompressed size, i.e. roughly 7 times smaller. In order to be downloadable the game shouldn't be bigger than 10 GB so that would give us 7 x 10 = 70 maps.

That doesn't sound like much at all, and we didn't even consider that music, sounds and voice acting need a lot of space too...

 

That said my calculation is probably completely wrong.

 

EDIT: Actually it sounds about right, though I guess the game could be bigger than 10 GB. I just hope it won't be too much, with my ****ty Wifi connection it takes ages to download 10 GB.

 

 

Honestly, I think you can get away with 25-30 gig for a download.

 

Storage is cheap, and ever increasing.  And bandwidth is cheap, and even regions that still sell service with caps have had significant increases in their caps over the last several years as broadband for media streaming has become expected customer behavior.

[Jarmo]

 

For a single level.  The number of levels in a dungeon isn't necessarily relevant, but the number of levels in the game is.  This is going to be a large game with a lot of maps.

 

 

EDIT: Actually it sounds about right, though I guess the game could be bigger than 10 GB. 

 

 

Largest game downloads right now are a bit over 30 GB.

50 GB for PE doesn't sound impossible, breaking 100 GB's would probably give some free press for being the biggast eva.

(but going past 50GB would likely start limiting purchases and I kind of hope it stays below that) 

[Grotesque]

This is not a console game.

 

Baldur's Gate had 5 CDs.

I don't mind  if Project Eternity will have 10 DVDs.

[coffeetable]

Image data can easily be compressed at 10:1 without a noticeable drop in quality (see the great applets here, and keep in mind that the .bmp versions would be ~4.5Mb), so if an uncompressed level is ~1Gb, then ~100Mb/map doesn't seem out the question. As well as that, the low-res textures have about a quarter of the pixel count of the high-res ones, so they should be ~25Mb/map.

[Sensuki]

they need a FLAC for image files

Edited September 14, 2013 by Sensuki

[Fearabbit]

Image data can easily be compressed at 10:1 without a noticeable drop in quality (see the great applets here, and keep in mind that the .bmp versions would be ~4.5Mb), so if an uncompressed level is ~1Gb, then ~100Mb/map doesn't seem out the question. As well as that, the low-res textures have about a quarter of the pixel count of the high-res ones, so they should be ~25Mb/map.

 

Two concerns:

1. The uncompressed files might have to contain multiple layers of information. For example, the alpha channel of (some of) the image files might be used to store information about the shininess of the surface. In that case you cannot achieve the same level of compression.

2. The pictures in the link already are very blurry. And it is not actually the gradients that cause problems, but the hard lines and edges between two colors. Especially screens that have very clear color schemes will show artifacts more easily.

 

The textures in Oblivion are compressed to 23% of their original size at best (i.e. the ones that don't contain additional information). I don't know what's possible in the engine Obsidian is using, but I wouldn't hold my breath for 10:1 compression.

Edited September 14, 2013 by Fearabbit

[coffeetable]

they need a FLAC for image files

A lossless compression format? There are loads, of which the most well known are probably PNG and lossless JPEG. They only manage 2:1 ratios though, so are only useful when the file is going to be edited somewhere down the line. If recipients are just looking at it, may as well use a lossy format with a high enough quality factor that no-one can tell. Edited September 14, 2013 by coffeetable

[coffeetable]

Two concerns:

1. The uncompressed files might have to contain multiple layers of information. For example, the alpha channel of (some of) the image files might be used to store information about the shininess of the surface. In that case you cannot achieve the same level of compression.

This shouldn't be a problem when you're designing the game. If you've got data that couldn't endure JPEG compression, pull it out to it's own file and stick it through lossless compression. That said, a lot of the non-art render data should be perfectly amenable to lossy compression.

 

2. The pictures in the link already are very blurry. And it is not actually the gradients that cause problems, but the hard lines and edges between two colors. Especially screens that have very clear color schemes will show artifacts more easily.

Are we looking at the same link? The suspension bridge has the hardest lines, and I don't see any artifacts appear until Q = 70, and that's a 300kb JPEG compared to a 4.5Mb bitmap.

 

e: Actually we know it's acceptable because the IWD image posted on the first page is a ~30Mb bitmap, but appeared on the game disk as a 4Mb JPEG.

 

The textures in Oblivion are compressed to 23% of their original size. I don't know what's possible in the engine Obsidian is using, but I wouldn't hold my breath for 10:1 compression.

Texture compression is a different beast to photograph (which is for our intents and purposes what PE's backgrounds are) compression. Textures are much smaller for one, but the big difference is that you need random, fast access to any point in the image.

 

I'm suspicious of a 23% reduction though. Any old texture compression alg should be able to get 2:1 at least. Are you comparing the compressed textures to losslessly compressed ones? Or ones that have simply been resaved in a lossless format?

Edited September 14, 2013 by coffeetable

[Fearabbit]

 

Two concerns:

1. The uncompressed files might have to contain multiple layers of information. For example, the alpha channel of (some of) the image files might be used to store information about the shininess of the surface. In that case you cannot achieve the same level of compression.

This shouldn't be a problem when you're designing the game. If you've got data that couldn't endure JPEG compression, pull it out to it's own file and stick it through lossless compression. That said, a lot of the non-art render data should be perfectly amenable to lossy compression.

 

2. The pictures in the link already are very blurry. And it is not actually the gradients that cause problems, but the hard lines and edges between two colors. Especially screens that have very clear color schemes will show artifacts more easily.

Are we looking at the same link? The suspension bridge has the hardest lines, and I don't see any artifacts appear until Q = 70, and that's a 300kb JPEG compared to a 4.5Mb bitmap.

 

e: Actually we know it's acceptable because the IWD image posted on the first page is a ~30Mb bitmap, but appeared on the game disk as a 4Mb JPEG.

 

The textures in Oblivion are compressed to 23% of their original size. I don't know what's possible in the engine Obsidian is using, but I wouldn't hold my breath for 10:1 compression.

Texture compression is a different beast to photograph (which is for our intents and purposes what PE's backgrounds are) compression. Textures are much smaller for one, but the big difference is that you need random, fast access to any point in the image.

 

I'm suspicious of a 23% reduction though. Any old texture compression alg should be able to get 2:1 at least. Are you comparing the compressed textures to losslessly compressed ones? Or ones that have simply been resaved in a lossless format?

 

 

1. No you can't. You can't pull out the alpha layer of an image. It stores just as much information as the actual image does and you can't take it away that easily. An image is nothing but data, if you do not compress that data then the overall size will be even bigger.

2. The bridge does not have "the hardest lines". If you paint a red circle on white background and save that with 60% JPG compression, you'll know what I'm talking about. Ugly, easily visible artifacts will appear even at comparatively lossless compressions.

3. Textures are a different beast, yes - you need a lot more of them in one single screen. So they need to be optimized even more, because otherwise your VRAM just won't be able to handle it. Which isn't a problem for Project Eternity.

4. 23% the size of the original image. Not 23% reduction. I'm talking about a 100:23 compression here, not 100:77.

 

These are things I know because I know a little bit about this stuff. For example I know that any compression is basically a Fourier transformation where the higher frequencies are cut off, resulting in a loss of detail especially when it comes to hard edges.

I also know that because of this, an image that uses a very small palette of colors (like the backgrounds from Icewind Dale, which are mostly dark or white and blue) will be much easier to compress than one that uses the full range of 24 bit of available colors.

 

What I do not know is: How the technology has advanced since 2006 and what kind of compression Project Eternity will use. Also whether there's actually any data stored in the alpha channel, whether they use normal maps, and how bad the artifacts actually are with the backgrounds they have created.

All I'm saying is that a compression to 10% the size is a bit optimistic, because it is purely based on the possible compression for a simple JPEG photography with lots of smooth edges. And because I've never seen a game use JPEGs as textures and I'm pretty sure there's a good reason for it.

[Sensuki]

yeah they use normal maps.

[coffeetable]

1. No you can't. You can't pull out the alpha layer of an image. It stores just as much information as the actual image does and you can't take it away that easily. An image is nothing but data, if you do not compress that data then the overall size will be even bigger.

Of course you can pull the alpha layer out. A bitmap is just an array with a cell for each pixel, and (for 32-bit colour) four bytes per cell (R, G, B and A). It's almost trivial to copy the RGB channels to a 24-bit image and the alpha to a separate 8-bit one.

 

A naive approach of lossily compressing the colour channels (24 bits/pixel @ ~10:1) and losslessly compressing the alpha channel (8 bits/pixel @ ~2:1) only buys you a 5:1 compression ratio, but that's a mile better than losslessly compressing the whole thing.

 

2. The bridge does not have "the hardest lines". If you paint a red circle on white background and save that with 60% JPG compression, you'll know what I'm talking about. Ugly, easily visible artifacts will appear even at comparatively lossless compressions.

They're the hardest lines you'll see often in photographs, which are the domain we're talking about (as they're the closest common analogue to PE's backgrounds). JPEG was designed for photo compression, and in that job it does exceptionally well. I don't know why you're even arguing this, because the IE games all used Q = 60ish JPEG backgrounds and they were fine.

 

3. Textures are a different beast, yes - you need a lot more of them in one single screen. So they need to be optimized even more, because otherwise your VRAM just won't be able to handle it. Which isn't a problem for Project Eternity.

I don't think you understand texture compression algorithms.

I also know that because of this, an image that uses a very small palette of colors (like the backgrounds from Icewind Dale, which are mostly dark or white and blue) will be much easier to compress than one that uses the full range of 24 bit of available colors.

 

These are things I know because I know a little bit about this stuff. For example I know that any compression is basically a Fourier transformation where the higher frequencies are cut off, resulting in a loss of detail especially when it comes to hard edges.

And you don't understand compression algorithms as a whole either. Some algorithms rely on DFT, but there are many other types too. And why would the IWD palette matter? For one most algorithms compressing >24 bit colour are ignorant about how muted the palette is (it's how "noisy" the image is that matters), and for two every IE game used JPEG backgrounds and got 10:1 compression ratios too, I just chose the IWD image because it was posted earlier in the thread.

 

What I do not know is: How the technology has advanced since 2006 and what kind of compression Project Eternity will use. Also whether there's actually any data stored in the alpha channel, whether they use normal maps, and how bad the artifacts actually are with the backgrounds they have created.

All I'm saying is that a compression to 10% the size is a bit optimistic, because it is purely based on the possible compression for a simple JPEG photography with lots of smooth edges. And because I've never seen a game use JPEGs as textures and I'm pretty sure there's a good reason for it.

The state of the art has advanced but everyone is still using JPEG anyway (because it's really good), the IE games used JPEG and got 10:1, and the IE games could use JPEG because backgrounds aren't textures. Edited September 15, 2013 by coffeetable

[Fearabbit]

 

Of course you can pull the alpha layer out. A bitmap is just an array with a cell for each pixel, and (for 32-bit colour) four bytes per cell (R, G, B and A). It's almost trivial to copy the RGB channels to a 24-bit image and the alpha to a separate 8-bit one.

 

 

Oh for the love of... yes of course you can do it, I was trying to say that it would create problems somewhere else. My point was that you can not take the alpha layer away, then have your 10:1 compression as if the data in the alpha layer suddenly disappeared. Like you said yourself you'd have to use 2:1 compression on the alpha layer, increasing the size of the compressed image. That's exactly what I said - you said 10:1 is possible, I said "ah, but there's an alpha channel that might cause problems".

 

Anyway, I have no interest in having this discussion with someone who, instead of saying "actually it works like this", prefers saying "apparently you have no idea what you're talking about, and I'm not gonna tell you".

I wasn't trying to take your 10:1 compression away, just saying I was skeptical about it.

[Hormalakh]

This is where multiple distribution networks would be great. Torrenting these games and then purchasing a license for each a la a game key or something. I don't think GOG will be able to serve that many people dling the game if it is that big.

[cdx]

This is where multiple distribution networks would be great. Torrenting these games and then purchasing a license for each a la a game key or something. I don't think GOG will be able to serve that many people dling the game if it is that big.

 

Good in principle. However, torrenting seems to be a very dirty word. A politically correct way to say it might be... "crowd-distributed"? Or supporter-uploaded? Backer-throttled? Please use my bandwidth, no, I'm not stealing the thing -ing? Maybe just plain old (and not as silly sounding) peer-to-peer.

 

P2P also gets the evil eye but if Blizzard can use it for their paid for games then it's probably allowed?

 

Come to think of it, Kickstarter ought to start some tracker for legal p2p distribution (without any mention of "torrent" of course). Kickstarted distribution. A legal tracker with a an integrated interface for licence key purchasing and all the support of the community should do quite well.

[Hormalakh]

Yep... "rhetoric always more rhetoric."

[J.E. Sawyer]

 

For the IE games, all levels were procedurally chopped up into series of 64x64 8-bit tiles and stored as .TIS files (IIRC).

[Chrząszczyrzewoszyczanin]

If requirements of Next CoD are anything to go by(50GB of free place on HDD), by the time Project: Eternity will be out game taking 20-30 GBs is not going to raise any eyebrows.

[Falkon Swiftblade]

I'm not sure if it's relevant here or not, but I came upon this article the other day that may help them save a ton of space. http://gamingbolt.com/granite-sdk-v2-0-now-available-reduces-ps4-pc-xbox-one-memory-usage-by-75-for-texture-streaming