Krycztij

TAW Moving Map Format

53 posts in this topic

I noticed I always get lost while flying in TFXplorer, so adding the moving map on the center MFD seemed like a good idea to

  • lower confusion
  • bring development forward.

Research on the moving map is pretty much scattered, with DKD having done the main work. So before I can even start implementing anything, I'll have to list all resources on the topic. What I found so far:

Until that point I thought of .lbm as a proprietary DID format, but turns out on Wikipedia it was one of the standard image formats on Atari, and it's 30 years old now :whoa: (don't know whether I should feel too young or too old)

  • TAW's files seem to be in the BMHD domain of the format.

That was the good news. The bad news is

  • it consists of interleaved bit planes …
  • … with run-length encoding.

So writing a decoder is not trivial any more and I surely won't do it tonight. Together with tiling and MFD infrastructure, it'll be a week of coding and I'll do it some time later.

This thread serves as a collection of resources on the topic, anyone feel free to add.

P.S.: Many pictures in the TAW Terrain Format thread are missing due to image hosters being run by idiots. (Was there really a need do delete 10 MB of images? A Wal Mart HDD can store millions of those, would it have been too much efford for you to buy one?) I keep archived copies of the pages, just write me if anyone needs them.

Share this post


Link to post
Share on other sites

You can save a bit of effort by converting the lbm file to bmp (or your favourite format) using some photo editor like Irfanview.

Here's some Python code to produce a composite 6400x6400 bmp of the whole map. It will come out upside-down as written, but it's lazier to flip it later. :)

# Read in all data from redse_.map into byte array
input=file("redse_.map","rb")
envmap=input.read()

# Read in all data from redse_.bmp into byte array 
#.bmp file converted from .lbm by irfanview,then flipped vertically. Must be exactly 321078 bytes.
input=file("redse_.bmp","rb")
mapbmp=input.read()

#Handle .env: Calculate the 160000 2-byte indices and put them in a list called envmap_index
envmap_index=[]
d=0
while d < 320000:
    g=ord(envmap[10+d])+ord(envmap[11+d])*256
    envmap_index.append(g)
    d=d+2

d=0
bmpheader=[]
while d<54:
    bmpheader.append(mapbmp[d])
    d=d+1

d=54
palette=[]
while d<1078:
    palette.append(mapbmp[d])
    d=d+1

d=1078
bmpbody=[]
while d<320000:
    bmpbody.append(mapbmp[d])
    d=d+1    

bigbody=[]       
y=0    
while y <400:
    xline=""
    yy=0
    while yy<16:
        x=0
        while x<400:    
            index=y*400+x #Map has a horizontal raster.....different to the ssd env
            tile=envmap_index[index]
            row=int(tile/16)
            col=tile-row*16
            base=(row*17*320)+(yy*320)+(col*17)
            xx=0
            while xx<16:                
                bigbody.append(bmpbody[base+xx])
                xx=xx+1
            x=x+1
        yy=yy+1    
    y=y+1           
        
outfile=file('tawbig.bmp','wb')
d=0
while d<54: #Rebuild the .bmp header for the larger size
    if d==2:
        outfile.write(chr(54))
    elif d==3:
        outfile.write(chr(4))
    elif d==4:
        outfile.write(chr(113))  
    elif d==5:
        outfile.write(chr(2))
    elif d==18:
        outfile.write(chr(0))
    elif d==19:
        outfile.write(chr(25))
    elif d==22:
        outfile.write(chr(0))        
    elif d==23:
        outfile.write(chr(25))          
    elif d==35:
        outfile.write(chr(0))          
    elif d==36:
        outfile.write(chr(0))   
    elif d==37:
        outfile.write(chr(113))  
    elif d==38:
        outfile.write(chr(2))          
    else:
        outfile.write(chr(ord(bmpheader[d])))
    d=d+1
    
d=0
while d<1024: # Reuse the palette
    outfile.write(chr(ord(palette[d])))
    d=d+1
    
d=0
while d<40960000:    
    outfile.write(chr(ord(bigbody[d])))
    d=d+1
This will only produce the base texture of course, you'll need to add the place names to an active map like TAW does. I think that information is in campaign.trg.

Share this post


Link to post
Share on other sites

Thanks — converting to BMP with a 3rd party tool would save most of the effort. But it would also mean the map runs only on my system :) I'll write a ILBM parser sometime, maybe it comes in handy with some other projects, too :)

Share this post


Link to post
Share on other sites

Sounds like you REALLY want the challenge of producing a parser. :)

I don't know if you've opened up the file yet, but it's a bunch of tiles that are selected to build up the map. Thus the same file can be used for Seaworld as the normal TAW map.

In contrast, EF2000 uses a single texture.

redse__1.jpg

...Ah, that's the upside down version.

Share this post


Link to post
Share on other sites

Well neither do I want to distribute 3rd party software with TFXplorer, nor do I want to distribute my own copy of the textures (not compatible to mods and huuuuge overhead for a 270 KB program). So there's only the parser left :)

Yes, DKD posted the texture in the TAW Terrain Format thread. I wonder if they assembled it on the CPU — GPUs at TAW's days probably couldn't handle textures that large …

Share this post


Link to post
Share on other sites

I doubt that the whole map is assembled in the game, probably just the area around the current position of the plane is used as the texture.

Share this post


Link to post
Share on other sites

I would certainly welcome any progress on solving the moving map problem. I'll have to pull out some of my old notes, yet, as I recall I spent a little time trying to modify both the BODY and CMAP chunks without solving it. The ILBM form is used frequently in TAW and yet the only problem with it, the moving map problem AFAIK, appeared at one of the old DirectX releases. Interestingly enough, there has not been much written or posted about this specific problem. It would be nice, if only to satisfy curiosity, to know what is the exact definition of the problem with reading the Interleaved data format in this moving map situation. Maybe we could get MS to fix it :rolleyes: Seriously though, perhaps the TAW fix would be simple if we understood the root of the problem. Looking forward to more tools :)

Share this post


Link to post
Share on other sites

I think the issue here is implementing the moving map in TFXplorer.

We know that redse_.lbm is a bitmap containing 814 16x16 pixel tiles, and redse_.map gives the index to these for each of TAW's 400x400 terrain tiles.

So, it's just a case of working out your position, creating a texture (say 256x256 which was the maximum for Glide) of the immediate environs. The texture then needs to be rotated to match the current heading, and some place name text added before displaying it on the MFD in a zoomable way.

I don't really see the problem. ;)

Share this post


Link to post
Share on other sites

Yes, the only problem is the ILBM format. Just gathering the facts here — so I don't have to search again when I come around to implement it :)

Share this post


Link to post
Share on other sites

I guess only a subset of ILBM is needed for the files that were used for TAW. I think it's use was more prevalent for the EF2000 GUI, so a parsing capability would be useful to have.

This has already been done in the Open Source world, eg here:

https://github.com/unwind/gimpilbm

so maybe the wheel doesn't need to be completely reinvented.

Share this post


Link to post
Share on other sites

Thanks Mike.

One of the discoveries with my analysis of redse_.ILBM in TAW is that it seems somewhat limited.

For example, there is, for obvious reasons, no CAMG chunk. The CAMG chunk contains expanded viewmodes, "HIRES" being one.

It would be nice to have the ability to support improved moving map graphics ;)

I do like the Jerry Morrison article on Wikipedia and you gentlemen may also find this article to be of value:

http://archive.org/stream/Amiga_ROM_Kernel_Reference_Manual_1987_Adn-Wesley_Publishing_Company/Amiga_ROM_Kernel_Reference_Manual_1987_Addison-Wesley_Publishing_Company_djvu.txt

Share this post


Link to post
Share on other sites

Thanks DKD. Wow, I've clipped out the relevant parsing chapter and ended up with 80kB of text.

If Krycztij hasn't already implemented it, I could try and produce pseudo-code for a minimum parser for our use...

Share this post


Link to post
Share on other sites

I found a (ADF?) file, noname11159, which appears to be one of Steve White's ILBM test files for TFX3. It is 320 x 200 and may be an early precursor to redse_.lbm which is 320 x 1000. It contains a number of chunks that report out as "UNKNOWN" in GWS, and therefore may be unique to TFX3. The DPPS chunk isn't even described in the lengthy Commodore Amiga documentation. Anyone know anything about ILBM DPPS chunks?

PS: Just for clarification, when I eluded to the "moving map problem", above, I was referring to the image corruption in the MFD in direct X rendered versions of TAW, not TFXplorer.

Share this post


Link to post
Share on other sites

No idea what a DPPS chunk is. I think anybody could make up a new name...

Finally got around to producing a parser in Python for redse_.lbm. This justs converts it to a palleted .bmp, for now. I should really try and produce an algorithm to produce a limited size 24 bit colour bitmap based on the position in game from this basic data.

# Read in all data from redse_.lbm into byte array
input=file("redse_.lbm","rb")
raw=input.read()

#palette starts at 48 (size 768), body starts at 824 (size 222557)
# Only useful info from the .lbm header is that size is 320x1000. RLE compressed with 8 bitplanes

i=824
j=0
tot_lin=0
tot_pix=0
buffer=[] #To contain de-RLE'd data
#RLE decode. Each line is compressed separately, and we have 1000 lines.
#Treat test character n as 'signed char'.
#If n is zero or positive, copy the next n+1 chars literally.
#If n is negative, copy the next char abs(n)+1 times.
while j<1000:
    while tot_lin<320:
        n=ord(raw[i])
        if n>128:
            n=n-256
        if n>-1 and n <128:
            t=0
            while t<n+1:
                buffer.append(raw[i+t+1])
                t+=1
            i=i+n+2
            tot_lin+=n+1        
        elif n==-128: #This never happens in redse_.lbm
            print 'NOP'
            i=i+1
        else:
            t=0
            while t<(n*(-1)+1):
               buffer.append(raw[i+1])   
               t+=1
            i=i+2
            tot_lin+=(n*(-1)+1)
    tot_pix+=tot_lin
    tot_lin=0
    j+=1

buffer2=[] #To comtain de-interleaved data
#Untangling the bitplanes is probably easier in C
#The first 40 bytes contain the 320 LSBs of the first line, the next 40 bytes, the BIT1s etc
x=0
y=0
while y<1000:
    while x<40:    
        z=(y*320)+x
        byte0=ord(buffer[z])
        byte1=ord(buffer[z+40])
        byte2=ord(buffer[z+80])
        byte3=ord(buffer[z+120])
        byte4=ord(buffer[z+160])
        byte5=ord(buffer[z+200])
        byte6=ord(buffer[z+240])
        byte7=ord(buffer[z+280])

        newbyte0=(byte7&128)+(byte6&128)/2+(byte5&128)/4+(byte4&128)/8+(byte3&128)/16+(byte2&128)/32+(byte1&128)/64+(byte0&128)/128
        newbyte1=(byte7&64)*2+(byte6&64)+(byte5&64)/2+(byte4&64)/4+(byte3&64)/8+(byte2&64)/16+(byte1&64)/32+(byte0&64)/64
        newbyte2=(byte7&32)*4+(byte6&32)*2+(byte5&32)+(byte4&32)/2+(byte3&32)/4+(byte2&32)/8+(byte1&32)/16+(byte0&32)/32
        newbyte3=(byte7&16)*8+(byte6&16)*4+(byte5&16)*2+(byte4&16)+(byte3&16)/2+(byte2&16)/4+(byte1&16)/8+(byte0&16)/16
        newbyte4=(byte7&8)*16+(byte6&8)*8+(byte5&8)*4+(byte4&8)*2+(byte3&8)+(byte2&8)/2+(byte1&8)/4+(byte0&8)/8
        newbyte5=(byte7&4)*32+(byte6&4)*16+(byte5&4)*8+(byte4&4)*4+(byte3&4)*2+(byte2&4)+(byte1&4)/2+(byte0&4)/4
        newbyte6=(byte7&2)*64+(byte6&2)*32+(byte5&2)*16+(byte4&2)*8+(byte3&2)*4+(byte2&2)*2+(byte1&2)+(byte0&2)/2
        newbyte7=(byte7&1)*128+(byte6&1)*64+(byte5&1)*32+(byte4&1)*16+(byte3&1)*8+(byte2&1)*4+(byte1&1)*2+(byte0&1)
        buffer2.append(chr(newbyte0))
        buffer2.append(chr(newbyte1))        
        buffer2.append(chr(newbyte2))
        buffer2.append(chr(newbyte3))
        buffer2.append(chr(newbyte4))
        buffer2.append(chr(newbyte5))       
        buffer2.append(chr(newbyte6))    
        buffer2.append(chr(newbyte7))        
        x+=1
    y+=1
    x=0
    
    
#Make as .bmp file as a sanity check....
buffer3=[] #To contain bmp file
#BITMAP FILE HEADER 14 bytes
buffer3.append(chr(66)) #B
buffer3.append(chr(77)) #M
buffer3.append(chr(54)) #bmp size 321078 bytes 0x0004e636, intel format and decimal= 54 230 4 0
buffer3.append(chr(230))
buffer3.append(chr(4))
buffer3.append(chr(0))
buffer3.append(chr(0)) #4 bytes of padding
buffer3.append(chr(0))
buffer3.append(chr(0))
buffer3.append(chr(0))
buffer3.append(chr(54)) #offset to image data 54 +1024=1078 or 54 4 0 0 decimal intel
buffer3.append(chr(4))
buffer3.append(chr(0))
buffer3.append(chr(0))
#BITMAP INFO HEADER 40 bytes
buffer3.append(chr(40)) #size of this header=40
buffer3.append(chr(0))
buffer3.append(chr(0))
buffer3.append(chr(0))
buffer3.append(chr(64)) #width 320 or 64 1 0 0
buffer3.append(chr(1))
buffer3.append(chr(0))
buffer3.append(chr(0))
buffer3.append(chr(232)) #height 1000 or 232 3 0 0
buffer3.append(chr(3))
buffer3.append(chr(0))
buffer3.append(chr(0))
buffer3.append(chr(1)) #colour planes = 1
buffer3.append(chr(0))
buffer3.append(chr(8)) #bits per pixel = 8
buffer3.append(chr(0))
buffer3.append(chr(0)) #compression method 0 0 0 0
buffer3.append(chr(0))
buffer3.append(chr(0))
buffer3.append(chr(0))
buffer3.append(chr(0)) #image size, 320000 or 0 226 4 0
buffer3.append(chr(226))
buffer3.append(chr(4))
buffer3.append(chr(0))
buffer3.append(chr(1)) #Horizontal rez, pixels per metre, set it to 1
buffer3.append(chr(0))
buffer3.append(chr(0))
buffer3.append(chr(0))
buffer3.append(chr(1)) #Vertical rez, pixels per metre, set it to 1
buffer3.append(chr(0))
buffer3.append(chr(0))
buffer3.append(chr(0))
buffer3.append(chr(0)) #Number of colours in Palette = 256
buffer3.append(chr(1))
buffer3.append(chr(0))
buffer3.append(chr(0))
buffer3.append(chr(0)) #Number of important colours, default to 0
buffer3.append(chr(0))
buffer3.append(chr(0))
buffer3.append(chr(0))

#Handle Palette, stored as BGRA in .bmp, RGB in .lbm
x=0
i=48
while x < 256:
    blue=raw[i+2]
    green=raw[i+1]
    red=raw[i]
    buffer3.append(blue)
    buffer3.append(green)
    buffer3.append(red)   
    buffer3.append(chr(0))
    i+=3
    x+=1
#Handle bitmap. Needs to be stored starting with the last line for .bmp    
y=999
while y > -1:
    x=0
    while x < 320:
        z=y*320+x    
        buffer3.append(buffer2[z])
        x+=1
    y-=1
        
check=len(buffer3)
print check  #should be 321078

outfile=file('redse_.bmp','wb')
x=0
while x < check:
    outfile.write(buffer3[x])   
    x+=1
outfile.close()

Share this post


Link to post
Share on other sites

My take on the ILBMs:

359ldnn.pngqs4dj7.png

arcade.lbm & clds2000.lbm

29msim8.png5z05lf.png

redse_.lbm & redsea.lbm

Does that match your results? Especially regarding the strange colors?

Share this post


Link to post
Share on other sites

That's about right. Apart from redse_.lbm, the only other file that might be useful is newmap.lbm which is a 800x800 map of the theatre.

Share this post


Link to post
Share on other sites

This is interesting. Would someone explain the relationship between the moving-map ILBM (redse_.lbm) and redsea.lbm, arcade.lbm, clds2000.lbm and newmap.lbm?

Thanks

Share this post


Link to post
Share on other sites

Seconding that. Are redsea.lbm / arcade.lbm legacy from the EF2000 system (with only one 400×400 bitmap for the whole scenario)? Why these colors?

I just implemented the moving map in TFXplorer, and it's taking an incredible amount of resources. Maximal sensor range is 200 NM, the top-left of the display is 280 NM from the F-22. Tile sidelength is 2.7 NM. That's a display size of up to 200×200 tiles, i.e. ~80,000 triangles and ~10 MB of vertex data!

I have no idea how they got that thing render efficiently in 2 MB of VRAM. It probably was a highly optimized software renderer. Maybe that's the reason they couldn't fix the moving map in D3D.

I could make it blazing fast with a specialized shader, but that would be a portability nightmare and a hassle for all API users.

Share this post


Link to post
Share on other sites

137-arcade-env-800.jpg

Thanks to mikew, we have corrected the names for the seaworld files in the ‘arcade’ group. There was not, however, a redse_map equivalent file hashed for the seaworld terrain. Many things in the TAW extracted files collection are not what they appear to be. They’ve hidden some things well; it’s like trying to find buried treasure. That is certainly a big part of the fun in figuring out how to assimilate the known components in order to solve the problems as the developers intended them to be. The solution to the absent ‘x_.map’ file is much like that for many of the TAW problems, the files for solving the problem are accessible, but figuring out how to utilize them to result in the correct solution is the challenge. The files to create the x_.map are the .4ev and .4e2 files. The files have to be used to create the mmap file by the following process: 1.) Flip and Rotate, 2.) effect a bitwise merge of the two files, which results in the creation of the 313kb .map file, 3.) add the appropriate header bytes. This results in the following:

Arc-mmap-gray.jpg

This file then communicates with a bitmaps file, which is in ILBM format, to produce the moving map displayed in the situation MFD.

The redsea world uses noname 1305 for its moving map tiles (redse_.lbm), noname 605 is the correct .lbm for the seaworld mmap tiles.

Curently,I am using the arcade.lbm color map for the game-interface briefing map, it works. I simply haven’t taken the time yet to convert it to a more aesthetically pleasing map.

There are a significant number of intriguing surprises about the seaworld world, many remain unexplored and I just need more time to work some of the bugs out and get the platform ready for continuing the Hardware adventure.

F222012-01-1123-19-28-06.jpg

I've got some questions here:

  1. One can generate seaworld's .map file from .4ev and .4e2. Is that also true for redse_.map? In this case, I'd throw away my .map handler and just always generate the maps from scratch, so I've got common code for Red Sea as well as Seaworld.
  2. What is noname605’s actual name? I've lost track of the "noname" table, sorry :(
  3. Your upper snapshot in the top-left shows tiles different from sea tiles, but in my version, I can only spot sea tiles. Was that a modification of yours, an error in the image, or do I have wrong game files?

Share this post


Link to post
Share on other sites
I just implemented the moving map in TFXplorer, and it's taking an incredible amount of resources. Maximal sensor range is 200 NM, the top-left of the display is 280 NM from the F-22. Tile sidelength is 2.7 NM. That's a display size of up to 200×200 tiles, i.e. ~80,000 triangles and ~10 MB of vertex data!

I don't understand why you need so many triangles. Isn't the moving map one textured quad, with the texture created on the fly?

A maximum range 100x100 tile map imples a 1600x1600 texture size though, and it clearly wasn't displayed like that in TAW.

Share this post


Link to post
Share on other sites

The maximal visible map is 200×200 tiles, i.e. 3200×3200 pixels, which is also 10 MB of palettized texture data (40 MB in true color), so this is no option either. (VRAM was 2 MB on average at the time of TAW's release, wasn't it?)

I strongly suspect they modified their software renderer to resolve the tile indices on-the-fly into rotated and scaled pixels.

Share this post


Link to post
Share on other sites

I've got some questions here:

  • One can generate seaworld's .map file from .4ev and .4e2. Is that also true for redse_.map? In this case, I'd throw away my .map handler and just always generate the maps from scratch, so I've got common code for Red Sea as well as Seaworld.
  • What is noname605’s actual name? I've lost track of the "noname" table, sorry :(
  • Your upper snapshot in the top-left shows tiles different from sea tiles, but in my version, I can only spot sea tiles. Was that a modification of yours, an error in the image, or do I have wrong game files?
1. .4ev and .4e2 are used in EF2000, TAW uses a single redsea.env file. It should be possible to generate a redse_map file from it or use the data directly. The little 16x16 tiles in redse_.lbm are just representations of the terrain. the 'raster' is different though.

2. I have noname605 as red1800\TEX_20.TM. The numbering system may be out by one though, but we don't know the names of noname604 and noname606.

Share this post


Link to post
Share on other sites

As I see it, there are three basic differences between redsea.env and redse_.map:

1. Header: size = 20 (45 4E 56 31 57 49 44 45 00 00 01 90 48 49 47 48 00 00 01 90) versus 10 (90 01 00 00 90 01 00 00 2E 03)

Format: Intel big endian versus little endian

2. Vertical raster versus horizontal raster

3. index pointer values

Frankly I don't recall all of what was modified when I constructed the seaworld MMap back then, I would have to dig out those notes. IIRC a correctly merged arcade.4ev with .4e2 yields the arcade.env.

The upper snapshot was done in hardware mode with hardware textures and the hardware T.O.D. palette, I believe that is the difference you are referring to.

We don't yet know the name of the 605.lbm, however, I have accumulated evidence to support my belief that it belongs to neither the default redsea world or the arcade / seaworld, but to a third world I have now been working on for some time ;)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now