How can they solve the massive texture problems?

[Frisk]

Even if you were unable to see a major difference with your eyes using lossy compression it becomes an issue when you start lighting special effects. This might introduce artefacts that you can see...

 

I really cannot see how that could be the case for static, pre-rendered backgrounds.. This might be an issue for textures on polygon models, but that's not what what we are talking about.

[stjuuv]

There's a lot of different estimates on the total size of the background images here, and a lot of them don't even elaborate on most of the assumptions they are making. Fortunately a few of the posters have already explained this as well, but everything depends on the amount of pixels you are expecting that the game backgrounds have in total, and the compression that can be applied to those images.

 

From this thread and a few other sources, it seems to me that lossless compression could get the file sizes down to about 4 bytes per pixel. Adding some lossy compression as well might maybe get the sizes down to even 2 bytes per pixel, but that is contestable.

 

Another question is the total amount of visible "screenfulls" of background images we are talking about. I would offer a ballpark figure of about 100 maps made up from 6 by 6 screens, so a total of 3600 about screenfulls of background images. That is quite a lot, but I am factoring in some larger exploration areas as well, which might not be as time-consuming production-wise as detailed city areas or complex indoors locations, but will nevertheless take up space on the disk.

So, using a native resolution of 2560x1440 pixels, we get a total background data of about 13 gigapixels. Using a 4 bpp compression, that translates to about 50 GB of image data, and using a more optimistic 2 bpp compression, about 25 GB of image data.

 

For a game that is likely released in 2014, I don't think that size (plus all the rest of the game data) should be a problem for anybody. The kickstarter was for not only a PST/BG/IWD style game, but also a good-looking game, and you can't expect a game that has good-looking (by modern standards) game with 2D-environments to be less than a DVD in size anyway.

Edited October 29, 2012 by stjuuv

[mstark]

I hope that the promised boxed editions of the game (which will have to be delivered on DVD's/BluRay's) wont make them hold back on game asset quality. Kinda dumb that they'll have to make the two versions, the physically delivered game very likely inferior to the downloadable one, if the game will be large.

[Frisk]

So, using a native resolution of 2560x1440 pixels, we get a total background data of about 13 gigapixels. Using a 4 bpp compression, that translates to about 50 GB of image data, and using a more optimistic 2 bpp compression, about 25 GB of image data.

 

uh...2bpp (bits per pixel) would mean 3.2GB, but that would probably mean image quality lower than a typical JPG file - Lossy compression is fine, as long as you don't go too low ... 8bpp would give us considerable space savings as well as image quality that would be VERY hard to distinguish from the original uncompressed one.

 

One other thing - we don't really need 32bpp for the original image - if the alpha channel is only used for transparency, it can essentially be just a monochrome map - meaning that the background only needs 25bpp.

 

The bottom line, however, is that depending on the compression quality and the image size, the background for a large area will take 50-150MB. Depending on the average areas size and the number of areas, we could come up with an estimate of the total space needed, but for now, the best guess is probably just "somewhere over 10GB".

 

Let's just wait and see...Obsidian might have some actual numbers in a few months.

Edited October 29, 2012 by Frisk

[stjuuv]

I hope that the promised boxed editions of the game (which will have to be delivered on DVD's/BluRay's) wont make them hold back on game asset quality. Kinda dumb that they'll have to make the two versions, the physically delivered game very likely inferior to the downloadable one, if the game will be large.

Does anyone have an idea of how much does it cost for publishers to order large batches of blu-ray discs or several DVDs with their "gone gold" game? Given that the flash memory prices are continuously dropping, then by spring of 2014 could it be feasible to use custom cased 64GB flash drives for distribution?

So, using a native resolution of 2560x1440 pixels, we get a total background data of about 13 gigapixels. Using a 4 bpp compression, that translates to about 50 GB of image data, and using a more optimistic 2 bpp compression, about 25 GB of image data.

uh...2bpp (bits per pixel) would mean 3.2GB, but that would probably mean image quality lower than a typical JPG file - Lossy compression is fine, as long as you don't go too low ... 8bpp would give us considerable space savings as well as image quality that would be VERY hard to distinguish from the original uncompressed one.

One other thing - we don't really need 32bpp for the original image - if the alpha channel is only used for transparency, it can essentially be just a monochrome map - meaning that the background only needs 25bpp.

I was using "bytes per pixel" not "bits per pixel" as units. Your value of 32 bits per pixel would therefore be the same as 4 bytes per pixel, and the numbers would work out the same as mine (close to 50GB of background image data), as long as we agree on the estimated amount of maps and sizes. Going to 25 bits per pixel would give a saving of about 20%, but would still keep the (relatively) lossless image total of tens of gigabytes.

The bottom line, however, is that depending on the compression quality and the image size, the background for a large area will take 50-150MB. Depending on the average areas size and the number of areas, we could come up with an estimate of the total space needed, but for now, the best guess is probably just "somewhere over 10GB".

Going by your figure of 25 bits per pixel, a 100MB image would be about 5800x5800 pixels or roughly 3 by 3 screen areas at a resolution of 2560x1440. Looking at BG2 (which I am playing right now), that is hardly a large area, especially at the native resolutions of BG2.

 

 

Most of the larger maps should be around 6 by 6 screen areas, meaning they should be around 4 times larger than your estimate, and a single background would land at about 400MB.

Edited October 29, 2012 by stjuuv

[Falkon Swiftblade]

I saw this article several months ago, this guys a frikken genius! He makes this whole scene with one texture map! http://www.3dworldmag.com/2012/04/02/one-texture-environment-created-with-udk/

[Frisk]

Going by your figure of 25 bits per pixel, a 100MB image would be about 5800x5800 pixels or roughly 3 by 3 screen areas at a resolution of 2560x1440. Looking at BG2 (which I am playing right now), that is hardly a large area, especially at the native resolutions of BG2.

 

Uh, no...because 25bpp is totally uncompressed data - which would just be plain silly. With lossless compression you should be able to easily get 15bpp - perhaps better. 8bpp (or one byte per pixel) is probably a reasonable compromise for good lossy compression without any visible loss in quality.

[mstark]

nvm

 

(apparently no way to delete posts?)

Edited October 29, 2012 by mstark

[Falkon Swiftblade]

It dawned on me today that Sony has already made mention that the PS4 will support 4k resolution and will likely come out in 2014 when this game comes out. This suddenly became a much bigger deal to me. If a Console system is going to support much higher resolution than this game, I do wonder what they're gonna do... here's one of a dozen such links. http://www.tgdaily.com/games-and-entertainment-features/65631-report-sonys-ps4-will-support-4k-resolution

[samm]

If it will support "4K" like the current PS supports 1080p, *yaaaaaaawn* is all I'm going to say about the PS4. One should never take a console as a benchmark for a PC anyway, plus it's completely irrelevant to the issue of how Obsidian is going to solve the storage issue they apparently face with this game.

[Starglider]

This is a fuss about nothing. Even really cheap and nasty GPUs in 2014 will be able to handle this stuff fine (relaively cheap ones do so just fine now). Two or three dual-layer DVDs should suffice (probably still cheaper than a dual layer blu-ray). Download caps keep rising for the digital delivery fans.

[Carmelle]

Just thought I'd throw some light on a few of the topics I've seen discussed, as I understand them. Feel free to skip down to the TL;DR Just wanted to add some insight.

 

On fractal-based textures: This is kinda cool, and it's however not too difficult to combine fractals to generate interesting looking textures for things like rocks, walls, brick, etc.. anything that's either regular and can be modeled mathematically (ex: bricks) or is very random (ex: sand). And you can always layer fractals on top of one another to add depth to the textures and make it difficult for the eye to detect any patterns or whatever. I used to do this a lot in my 3d work back in the day, i.e. procedurally generated textures. Placement of things could be calculated once on level load, and you switch out CPU time/ram usage for storage space.

 

The problem with this is it takes some control away from the artists, and kind of goes against (what I think is) the design ethos of the project: to create beautifully hand drawn (or at the very least, hand placed) worlds, full of detail, variety and purpose.

 

On the topic of compression: Generally lossy formats like JPEG don't provide a simple and hard-set conversion in terms of, if the original picture is x bits per pixel it'll compress down to y bits per pixel or whatever. From what I remember in messing with lossy encoders, there's a few different tricks you can follow. 1) One is encoding parts of the images as a composition of other parts of the image. 2) Another is doing heavy down sizing to different color or chroma channels that the eye can't easily distinguish (ex: blue/yellow). 3) And the third is replacing color information in the picture with an index. You develop an index of all colors used in the picture and assign each a number, then each pixel only has to store x,y,color-index instead of x,y,r,g,b and the index has a defined color per ID in only one spot per image (This is similar to what most loss-less formats probably do, along with other tricks I'm sure). 3b) You can then do things like, combining colors that are visually similar together so you remove some colors from the image too. The first two and 3b are what lead to things like artifacts appearing in the image.

 

So what this means is that the amount of compression you get varies greatly depending on the entropy of the image itself. Lots of different colors near each other with very little stretches of one or similar colors = bad compression. But if an image uses lots of similar colors and areas where one pixel easily blend into another (like the IWD pic used earlier), it's easier to compress more. Just something to keep in mind with the estimations.

 

And lossy compression is definitely viable for a game, you just wouldn't want to push it too far.

 

On the topic of actually using/displaying the massive environments: that's much easier. They can be drawn in, and separated into, smaller chunks. Then you could preload blocks in the direction your character is walking, before they're rendered onto the screen. No need to have an entire aaaaaaaaa x yyyyyyyy 500 meg in memory at once. More likely though, it'll probably be some sort of 3d asset(s) that have textures applied to them. A building could be actually rendered in 3d and have a very detailed texture applied to it. Especially since they said that environments were being created in 3d first, and then modified in some sort of post process. Plus doing it this way would allow them to keep extra information that could be useful for pathing, occlusion, dynamic lighting, etc...

 

And as for time it'd take to create the environments: I suspect there will be duplicated assets (either drawn in as some sort of brush, or actually placed as a separate entity within the game) all over the place to make it less work. Not *everything* needs to be unique. And by keeping them as separate entities, it could help reduce disk usage. 5 trees could be used 5 times each to make a small group, instead of drawing and keeping texture information for 25 separate ones.

 

###

 

TL;DR:

 

Fractals are cool, but detract from artists ability to really make the world their own.

 

As for compression, it's hard to speculate on numbers without knowing more about the worlds. So really it just feels kinda pointless to me... Finding solutions and stuff might help, but things like EXR are already well known and used in 3d work, and I'm sure these devs know of a lot more solutions too. I'm not saying there's no point in discussing, I enjoy it Just saying, not sure how much we can offer the devs short of finding some magic and miraculous new way of doing it that most people have never done before But hey, if you do come up with something, lemme know, we'll go sell it

 

Display/Creating massive environments, I'm pretty sure they have that on lock if nothing else. No worries

Edited October 31, 2012 by Carmelle

[stjuuv]

Going by your figure of 25 bits per pixel, a 100MB image would be about 5800x5800 pixels or roughly 3 by 3 screen areas at a resolution of 2560x1440. Looking at BG2 (which I am playing right now), that is hardly a large area, especially at the native resolutions of BG2.

Uh, no...because 25bpp is totally uncompressed data - which would just be plain silly. With lossless compression you should be able to easily get 15bpp - perhaps better. 8bpp (or one byte per pixel) is probably a reasonable compromise for good lossy compression without any visible loss in quality.

I already covered that in my original post as well. Your estimate of 15 bits per pixel for lossless compression is pretty much the same as my estimate of 2 bytes per pixel, and it would still add up to about 25 GB of raw image data. If you are optimistic enough to expect a 1 byte per pixel average compression rate (which would, in my opinion, indicate a rather dull and colorless world, which I don't think will be the case) it would still add up to more than 10 GB of raw background image data without any other game content, which is likely to take a considerable amount of space as well, now that we are going for orchestrated soundtracks etc etc.

That is more than 1 (closer to 1,5) dual-layer DVD worth of data just for the images, and is still something that would need to be taken into account when starting to distribute the game. I personally think the image data might be more than 2 dual-layer DVDs in itself, plus everything else. That will raise the question of whether it is desirable to distribute the total game on 3+ DVDs, or consider using other options for physical media.

Edited October 31, 2012 by stjuuv

[SteveB]

Sorry I'm just jumping in here at the end, but I thought some of Allegorithmic's tools might be worth looking into:

 

http://www.allegorit..._pages/products

 

They've got some texture reduction middle-ware that can be run with Unity, like Substance Redux which can be used even after a game is completed to reduce texture sizes. I don't know how applicable it is to Project Eternity, but I thought it was interesting and maybe worth checking into. This article's a few years old, but it talks a bit about their 'procedural texturing' process:

 

http://www.bit-tech....res_Future_Gam/

 

The way I see it, Substance Redux could be used post development to reduce the file size of the game, which could be helpful for distribution. And Substance Air could be used to reduce load times, perhaps, by procedurally generating the textures at run time.

 

No idea though.

 

Cheers.

[Ynk]

Those are procedual textures, they are smaller in size since they are defined by mathematical functions, rather than pixels. Procedual textures are usefull in a realtime 3d-engine, or might be the prefered workflow for an artist working in a 3d-software (for example since they are animateable). But once the artist renders the scene, the size of the resulting bitmap will not differ.

[Luridis]

I just love how I got all kinds of crap at the very mention of tiles in this thread. What are they using in the most recent update: tiles.

[Frisk]

I just love how I got all kinds of crap at the very mention of tiles in this thread. What are they using in the most recent update: tiles.

Totally different thing.   The game is not using tiles - Obsidian is using tiles to build backgrounds, which will then be modified by hand.

[Luridis]

I don't see it that way... They're still making something from a tile set which means using tiles in itself doesn't equal bad graphics, which was my point. I think the IE games used the 1 piece backgrounds because, when they came out, 3D modeling & texturing was still in its infancy. Tiles can look good, provided there is enough variety and they're well done. Hopefully there will be a way for modders to use them.

[Osvir]

Does anyone really think something like this could be done using tiles?

 

 

No, but this can:

[Hassat Hunter]

Using said process still doesn't allow "random-generated" maps like you want for 'tiles'.

 

Not that I see why anyone wants that above beautifully handcrafted terrains still...

[Osvir]

Compare handcrafted beautiful-ness with tiles. There's one factor why I think tiles should be considered:

Working with tiles, at least for dungeons, allows Obsidian to do more dungeon.

I'd love to have as many levels/areas as possible (not to mention the 15 level mega dungeon) and asking Obsidian to do everything handcrafted feels like a lot of pressure. Hey I say, if they can make a lot of areas handcrafted (without tiles) then that'd be awesome. But with tiles it allows Obsidian to do so much more, and fan modders as well.

But then, here comes the question, "Quality over Quantity". With Quality you get (example) 10 levels, with tiles/Quantity you can get 20 levels. Then it strikes me that an IE spiritual successor would be the latter, Quality would be to have Quantity (More Levels/Tiles > Less Levels/Handcraft). I absolutely love the drawing there, but if Tiles allows Obsidian to work faster and create more content, then I say (personal opinion) that it might be a reason to at least consider Tiles for Dungeons (which is what Obsidian is already doing, I'm just advocating for it further ).

On Random Generation+Tiles, conceptually: As long as you have 4 corners and 4 walls, if you know the math, then you could just press "Enter", compile, and you'd get a random generated dungeon every time.

Tiles = Less workload
Handcraft = More workload

It took Obsidian 1 month, approximately, to release the screenshot. Taking that into consideration, it might take 1/2 month for a screenshot to manifest (what with deadline and all). Tiles allows Obsidian to give us an abundant amount of areas, whilst handcrafts would give us about 36 areas. That is if they make 2 beautiful areas a month.

My point: Dungeons, Catacombs, Houses, Corridor-y stuff: Make it into tiles. Save some time. If it saves time.
More open overland areas (Forest, City, Mountain etc. etc.) = Handcraft.

[Pipyui]

Ah, I think I may have totally misunderstood this thread before.  My understanding was that many were advocating the use of tiles as in the vein of NWN1 - as in reducing map "sizes" by replacing them with big texture grids to be rendered on the fly.  No, it doesn't particularly surprise me that dungeons and caves and such are to be modeled using prerendered tiles as like Oblivion, not as in NWN1 (though I should hope the variety to be much improved over Oblivion's caves and dungeons).

 

*shrug* Though really, I might just be unconsciously altering my old opinion to fit with new information so as to imagine I was correct.  I honestly don't remember anymore (it's really late, and I'm too tired to look into it).

Darn you brain!  You're always like a closed book!

[Jarmo]

With jpeg a 20:1 compression ratio is pretty much indistinquishable from the original,

50:1 and you start seeing the color degradation and artifacts.

 

I wouldn't expect the background image storage size to be any major hurdle in this.

[Falkon Swiftblade]

I posted about this on another thread, but theres a new industry standard for compression that just came out called HVEC or H.265 & will support streaming 8k resolution over the internet. It can currently stream 4k at 10Mbits, vs 25Mbits of 1080p at the highest quality now. http://www.hdtvtest.co.uk/news/ntt-docomo-hevc-4ktv-201302262701.htm

[gertiyolem]

they can solve it in the long run, amidst the clamor of gamers