Jump to content
COMBATSIM Forum
Krycztij

TAW terrain format

Recommended Posts

Thanks a lot, Mike! The wheels look much larger in any picture on the internet, so I’ll treat it as a flaw in TAW’s shape! Those engine pods look small as well.

Share this post


Link to post
Share on other sites
20 hours ago, DrKevDog said:

This is my current list of worlds:

lev\redsea.env
lev\redsea.4ev
lev\arcade.env
lev\arcade.4ev
lev\newworld.4ev
lev\norway.4ev
lev\iceland.lev
lev\korea.lev
lev\angola.lev
lev\atlantic.lev
lev\craig.lev
lev\clouds.lev
lev\faeroe.lev
lev\hasbro.lev
lev\libya.lev
lev\martin.lev
lev\nenorway.lev
lev\nwnorway.lev
lev\senorway.lev
lev\swnorway.lev
lev\norway.lev
lev\wales.lev
lev\world.lev
lev\island.lev

There is perhaps one more, name unknown.

It might now be useful to categorize them according to their project affiliations. most of the lev / le2 worlds are from the tfx2 project, another group appears to be from the fws.pc (Fighter Weapons School) project and the Island lev / le2 is associated with the TIALD Military Simulation project (at least that is what the .asc file suggests).

The island has some interestingly unique features. For example: In this pixel image of the Island, the 2 large white areas represent 4 world space tiles which have pointer values of "FFFF" and I suspect that has something to do with TIALD targeting systems, but needs further analysis. Five tiles code unusually for ships, but they have also not been discovered.

 

Island-FE-2.jpg.e4f4d41aa072078b7df73c1bbcc0c810.jpg

Soon as I discover the hidden TIALD encoded military secrets that D.I.D was working on, you won't be hearing or seeing much from me :o

No! We need you! :D

Regarding FFFF: In TFX2 as well as TFX3, FFFF marks a missing terrain tile. I.e. a hole in the terrain. No shape is found there.

These are rare to find, but there must have been some of them in unused worlds, otherwise I wouldn’t have programmed it like that. (I remember specifically, that I didn’t want that special case in my TFX2 handling but then some worlds would just crash TFXplorer so I had to insert it.)

If you find a better meaning for FFFF, I’ll be very happy to implement it. But from what I’ve seen, it’s just a placeholder for void. (Other games have them as well, e.g. the tile maps of Ace Combat 2 & 3).

 

Share this post


Link to post
Share on other sites

After a gap of a few months, I'm again trying to see if there's an easy way to extract relevant information from the .3 files and and repackage it into VRML or whatever.

The starting point is a set of text files I created a couple of years ago where I separated all TFX3's .3 files into LODs and metadata like this:

antenna_meta.png.1c8cb35f212131c6ac5a384be11d5d98.png

Now, I just need some 'machine' to plough through the bytecode and magically 'know' what is relevant and what can be discarded.

In the above example of the most distant LOD of antenna.3, there are only 2 vertices but an awful lot of bytecode dealing with them. While I can manually look at that and say we probably don't need any of this at all, I don't want to be doing that for all 2000 or so models.

 

I did start making a Python script to decompile the bytecode to work out all possible paths through the model, but the logic was getting out of hand.

Now I'm wondering if I should be seeking inspiration in the 3View code instead.

 

Does anybody have any thoughts on how to do this?

 

Share this post


Link to post
Share on other sites

My thoughts on this:

  1. Use the distance markers to put the visuals into some kind of mapping. I.e. if you encounter a “display if closer than 2000 units” marker, all later visuals go to the “2000 or closer” category, until interrupted by another marker. In the end, store each bin into VRML’s “LOD” node.
  2. I’d probably gather all triangles from all paths. In the end, I’d try to find duplicates and remove them. (Since TFX uses integer coordinates, it shouldn’t be prone to rounding errors.) I don’t know about conflicts; I’d probably go for the most detail then (i.e. if a flat-shaded triangle overlaps a textured one, choose the textured and discard the flat-shaded one). Though TFX sometimes uses overlapping transparent triangles as a mean of applying color gradients to textures; I guess there’s something like that on the Aswan dam but I’m not sure.
  3. Throw away all shadows. IIRC, they make up a *huge* portion of the bytecode.

Regarding 3View: Sort the source code by file size; the interpreter is one of the largest files. But it may be hard to use, because I pre-translated the bytecode and merged some instructions (e.g. triangle vs quad rendering) due to the extreme performance requirements …

Share this post


Link to post
Share on other sites

Thanks! After writing that, I took a long walk to get some lunch, did some thinking on the way and came to about the same conclusion as you.

 

It's probably better to start from scratch and not try to 'polish the turd' of my previous attempt. 😐

 

Share this post


Link to post
Share on other sites

A week later and not much to report...

 

At the moment I'm only using the 3View code for the excellent comments on what the opcodes do.
In order to break this down into smaller tasks, I'll make multiple passes through the bytecode which 3View doesn't do.

 

The hardest task is probably getting a list of all possible paths through the bytecode then culling the irrelevant ones in some evaluation step.

 

So far, I've only implemented the simplest case of a basic terrain tile but even this shows up a couple of things to deal with.
The only real decision is whether the distance further than 6000 or not (0027 opcode), so this leads to just two paths which in this case are LODs.
So if I then 'run' the bytecode with the two cases, I encounter 0015 (rendering order) opcodes in the highest LOD case. As is suggested above, it's best just to extract all the polygons from 0015-TRUE and 0015-FALSE and sort them out in the next step. There are also vertices that are only used for the 0015 decisions, so these need to be culled as well.
The other LOD just consists of shaded triangles, but this brings up a time of day problem. The triangle colour depends on the palette, so I'll probably create a different set of files for each time of day. I'm going to have to do a similar thing for the palette/texture combinations anyway by producing a set of BMP files.

 

I've spent most of the limited time I have for TFX setting up a new development environment though. Since I'm using LInux more, I'm finding that Microsoft's VS Code is good in that it looks and works the same on Windows and Linux. That can't be said of my Python code though. It took a while to get the same output on various combinations of Win7/win10/Linux/Python2/Python3/32bit/64bit. The problem was mostly line endings and not putting my print statements in brackets, but Python3 seems to think 5/2=2.5 and not 2.

Share this post


Link to post
Share on other sites

I'm glad you are still working on it. Off the top of my head I seem to recall there was a convention to place the hyperplane bytecode in a easily accessible locale, like at the end of the block.

That would make 0015 culling fairly easy. I will have to check on that though...

 

Tile Hyperplane key:

 

663803254_0015-New-Apex400-Copy.png.1f6735091f21ce08ec0417352b295ac0.png

 

 

Share this post


Link to post
Share on other sites

I don't think there is too much else hidden in the .3 file itself, but we may not understand exactly how the information is used. For instance, the LOD2 shaded quads are described like this:

00020018001700000007
0071000000170001

We've interpreted this as:

1. Draw a flat shaded triangle using palette colour 0x18 between vertices 0x17,0x00 and 0x07

2. Keep 0x18 as the colour, but draw a second triangle at 0x00, 0x17 and 0x01

 

Now when we fly around, we don't see a constant colour for LOD2 but something more dynamic. That may be because the engine is doing something extra with the 0071 opcode, but I'm not going to worry about that for this exercise.

I forgot that the 0015 lines are nested which gives quite a complex behaviour, but again I'm trying not to care about that except to navigate through all paths and suck up all polygon descriptors.

 

Hmmn, I'm going to need to check that after I've handled a .3 file that all lines in the file have been 'touched' by the analysis code to avoid missing anything.

Share this post


Link to post
Share on other sites
10 hours ago, mikew said:

Now when we fly around, we don't see a constant colour for LOD2 but something more dynamic.

 

Yes, this type of quads/triangles is subject to fogging. TAW has at least two fog modes – hardware-accelerated and software-based – depending on a file being present in the root directory; I forgot its name. 128.nvidriva or something like that. Long story short: You had to place that file there if you had a certain kind of GPU to make use of hardware fogging.

 

Keep in mind that in TFXplorer, anything is subject to atmospheric fog, so the distinction is useless here.

Share this post


Link to post
Share on other sites
On 9/10/2018 at 4:14 AM, mikew said:

Now when we fly around, we don't see a constant colour for LOD2 but something more dynamic.

 

I think 0076 is a very sneaky and elusive opcode / flag and the full definition remains a bit FOGGY...

Is it possible for you to post a couple of images?

Share this post


Link to post
Share on other sites

Right now, I'm ignoring those opcode/flags at the start of the file, like these from arablm_1.3:

0000; 00930004    ; If 0093 flag set, jump to line 2
0001; 0000    ;
0002; 002717700008    ; If Distance >6000 then jump to line 5
0003; 007b0004    ; If 007b flag set, jump to line 5
0004; 0000    ;
0005; 00760004    ; If 0076 flag set, jump to line 7
0006; 0000   

 

I'm only dealing with the vertices and polygons for the time being.

Something is going extremely wrong with my texture coordinates (or their relationship with the vertices) , but I have no idea what.

texwrong.png.45ae0970c6933ec1a0f38845baa6ae6b.png

Share this post


Link to post
Share on other sites

I guess you posted a good one yourself :)

 

In TFXplorer, it doesn’t matter. The polygon is just flat-shaded in the constant color, but you don’t see the anomaly because it’s too far away.

 

By the way, I don’t think we should care about terrain LOD here – remember how I rendered all 40,000 tiles brute force in 2011? :) 

2011-06-12postedimagexxe73.jpg

 

Just throw away the low-detail versions :) 

Share this post


Link to post
Share on other sites
13 minutes ago, mikew said:

I'm only dealing with the vertices and polygons for the time being.

Something is going extremely wrong with my texture coordinates (or their relationship with the vertices) , but I have no idea what.

texwrong.png.45ae0970c6933ec1a0f38845baa6ae6b.png

 

Interesting result. Could the 4th UV coordinate be wrong? For a general guideline, check TFX Shape executeBytecode.cpp and look for a variable named uvs – all texture coordinates pass through it and as far as I remember, they’re separate from vertices.

 

Edit: From looking at you screenshot I’m quite sure you have assigned one UV for each vertex. That’s wrong; adjacent triangles share their vertices but not their texture coordinates! Whenever you encounter a triangle or quad, you need to actually generate a completely new vertex with the UV in a separate vertex list and not share it with other triangles/quads than the one you’re current encountering. Your resulting list will almost always have more vertices than the 3 file. If you want to remove duplicates, do so after having parsed the entire file and also decide vertex equality by UV, not just by position!

 

(I think this is the reason why I haven’t made TAW’s smooth polygon shading work. I just can’t identify which triangles share their vertices without serious complexity/performance cost in the parser.)

Share this post


Link to post
Share on other sites

I'm only dealing with triangles, so just have 3UV pairs. For a simple terrain tile like arablm_1.3, I'm just collecting the vertex information and triangle descriptors into an intermediate format like this:

 

 


[-1024, -100, -1024]
[-512, -100, -1024]
[0, -100, -1024]
[512, -100, -1024]
[1024, -100, -1024]
[1024, -100, -512]
[0, -100, -512]
[-1024, -100, -512]
[-1024, -100, 0]
[-512, -100, 0]
[0, -100, 0]
[512, -100, 0]
[1024, -100, 0]
[1024, -100, 512]
[-1024, -100, 512]
[0, -100, 512]
[0, -100, 1024]
[-1024, -100, 1024]
[-512, -100, 1024]
[512, -100, 1024]
[1024, -100, 1024]
[512, -120, 512]
[-512, -120, 512]
[-512, -120, -512]
[512, -120, -512]
[512, -1000, -512]
[-512, -1000, -512]
[-512, -1000, 512]
[512, -1000, 512]
[0, -1000, 0]
002e0000000300470000002f0017002f0000002f00030013000f0010
002e00000003002f00170047000000470017002f0003000f00130015
002e00000003005f00000047001700470000002f0003001400150013
002e0000000300470017005f0000005f0017002f000300150014000d
002e0000000300470017002f002f002f0017002f00030015000a000f
002e00000003002f002f004700170047002f002f0003000a0015000b
002e00000003005f00170047002f00470017002f0003000d000b0015
002e000000030047002f005f0017005f002f002f0003000b000d000c
002e00000003000000000017001700000017002f000300110016000e
002e00000003001700170000000000170000002f0003001600110012
002e0000000300170000002f001700170017002f00030012000f0016
002e00000003002f001700170000002f0000002f0003000f00120010
002e00000003000000170017002f0000002f002f0003000e00090008
002e000000030017002f0000001700170017002f00030009000e0016
002e0000000300170017002f002f0017002f002f00030016000a0009
002e00000003002f002f00170017002f0017002f0003000a0016000f
002e00000003002f002f00470047002f0047002f0003000a00180006
002e0000000300470047002f002f0047002f002f00030018000a000b
002e000000030047002f005f004700470047002f0003000b00050018
002e00000003005f00470047002f005f002f002f00030005000b000c
002e00000003002f00470047005f002f005f002f0003000600030002
002e000000030047005f002f004700470047002f0003000300060018
002e0000000300470047005f005f0047005f002f0003001800040003
002e00000003005f005f00470047005f0047002f0003000400180005
002e000000030017002f000000470000002f002f0003000900070008
002e00000003000000470017002f00170047002f0003000700090017
002e00000003002f002f001700470017002f002f0003000a00170009
002e0000000300170047002f002f002f0047002f00030017000a0006
002e00000003001700470000005f00000047002f0003001700000007
002e000000030000005f001700470017005f002f0003000000170001
002e00000003002f00470017005f00170047002f0003000600010017
002e000000030017005f002f0047002f005f002f0003000100060002

 
 

 

 

Then knowing the texture is tex_2 from the header, we should have enough information to make a VRML file for the 32 triangles.

 

 


#VRML V2.0 utf8

Shape {
appearance Appearance {
texture ImageTexture {
url "..\red1400bmp\tex_2.bmp"
repeatS FALSE
repeatT FALSE
}

}

geometry IndexedFaceSet {

coord Coordinate {
point [
-1024 -100 -1024,
-512 -100 -1024,
0 -100 -1024,
512 -100 -1024,
1024 -100 -1024,
1024 -100 -512,
0 -100 -512,
-1024 -100 -512,
-1024 -100 0,
-512 -100 0,
0 -100 0,
512 -100 0,
1024 -100 0,
1024 -100 512,
-1024 -100 512,
0 -100 512,
0 -100 1024,
-1024 -100 1024,
-512 -100 1024,
512 -100 1024,
1024 -100 1024,
512 -120 512,
-512 -120 512,
-512 -120 -512,
512 -120 -512,
512 -1000 -512,
-512 -1000 -512,
-512 -1000 512,
512 -1000 512,
0 -1000 0
]
}
texCoord TextureCoordinate {
point [
0.27734375 0.0,
0.18359375 0.119791666667,
0.18359375 0.0,

0.18359375 0.119791666667,
0.27734375 0.0,
0.27734375 0.119791666667,

0.37109375 0.0,
0.27734375 0.119791666667,
0.27734375 0.0,

0.27734375 0.119791666667,
0.37109375 0.0,
0.37109375 0.119791666667,

0.27734375 0.119791666667,
0.18359375 0.244791666667,
0.18359375 0.119791666667,

0.18359375 0.244791666667,
0.27734375 0.119791666667,
0.27734375 0.244791666667,

0.37109375 0.119791666667,
0.27734375 0.244791666667,
0.27734375 0.119791666667,

0.27734375 0.244791666667,
0.37109375 0.119791666667,
0.37109375 0.244791666667,

0.0 0.0,
0.08984375 0.119791666667,
0.0 0.119791666667,

0.08984375 0.119791666667,
0.0 0.0,
0.08984375 0.0,

0.08984375 0.0,
0.18359375 0.119791666667,
0.08984375 0.119791666667,

0.18359375 0.119791666667,
0.08984375 0.0,
0.18359375 0.0,

0.0 0.119791666667,
0.08984375 0.244791666667,
0.0 0.244791666667,

0.08984375 0.244791666667,
0.0 0.119791666667,
0.08984375 0.119791666667,

0.08984375 0.119791666667,
0.18359375 0.244791666667,
0.08984375 0.244791666667,

0.18359375 0.244791666667,
0.08984375 0.119791666667,
0.18359375 0.119791666667,

0.18359375 0.244791666667,
0.27734375 0.369791666667,
0.18359375 0.369791666667,

0.27734375 0.369791666667,
0.18359375 0.244791666667,
0.27734375 0.244791666667,

0.27734375 0.244791666667,
0.37109375 0.369791666667,
0.27734375 0.369791666667,

0.37109375 0.369791666667,
0.27734375 0.244791666667,
0.37109375 0.244791666667,

0.18359375 0.369791666667,
0.27734375 0.494791666667,
0.18359375 0.494791666667,

0.27734375 0.494791666667,
0.18359375 0.369791666667,
0.27734375 0.369791666667,

0.27734375 0.369791666667,
0.37109375 0.494791666667,
0.27734375 0.494791666667,

0.37109375 0.494791666667,
0.27734375 0.369791666667,
0.37109375 0.369791666667,

0.08984375 0.244791666667,
0.0 0.369791666667,
0.0 0.244791666667,

0.0 0.369791666667,
0.08984375 0.244791666667,
0.08984375 0.369791666667,

0.18359375 0.244791666667,
0.08984375 0.369791666667,
0.08984375 0.244791666667,

0.08984375 0.369791666667,
0.18359375 0.244791666667,
0.18359375 0.369791666667,

0.08984375 0.369791666667,
0.0 0.494791666667,
0.0 0.369791666667,

0.0 0.494791666667,
0.08984375 0.369791666667,
0.08984375 0.494791666667,

0.18359375 0.369791666667,
0.08984375 0.494791666667,
0.08984375 0.369791666667,

0.08984375 0.494791666667,
0.18359375 0.369791666667,
0.18359375 0.494791666667
 ]
}
coordIndex [
19 15 16 -1,
15 19 21 -1,
20 21 19 -1,
21 20 13 -1,
21 10 15 -1,
10 21 11 -1,
13 11 21 -1,
11 13 12 -1,
17 22 14 -1,
22 17 18 -1,
18 15 22 -1,
15 18 16 -1,
14 9 8 -1,
9 14 22 -1,
22 10 9 -1,
10 22 15 -1,
10 24 6 -1,
24 10 11 -1,
11 5 24 -1,
5 11 12 -1,
6 3 2 -1,
3 6 24 -1,
24 4 3 -1,
4 24 5 -1,
9 7 8 -1,
7 9 23 -1,
10 23 9 -1,
23 10 6 -1,
23 0 7 -1,
0 23 1 -1,
6 1 23 -1,
1 6 2 -1
 ]
}

}
 

 

 

Obviously, I'm missing something. :angry:

 

EDIT: Just noticed your edit. That would explain it...

Share this post


Link to post
Share on other sites

This may be a silly ask...but did you intentionally include the hyperplanes ?

50 minutes ago, mikew said:

512 -1000 -512,
-512 -1000 -512,
-512 -1000 512,
512 -1000 512,
0 -1000 0

 

Share this post


Link to post
Share on other sites

No, they are there because I was too lazy to remove them. I have removed the duplicate polygons from that intermediate data though.

It seems that I'm going to have to rebuild the vertex list anyway. I'll remove them then.

Share this post


Link to post
Share on other sites

For the 32 triangles making up the tile, we now have 96 geometry vertices so the triangles are completely independent.

I've left the coordinate system as it is, and I get the impression that the texture is being applied to the wrong side. :)

ctclfs_1.png.00269a6230325dde016f21281a3f4418.png

Share this post


Link to post
Share on other sites

Awesome!


I have a feeling you forgot to negate the vertical coordinate :) (I think Y points towards the ground in TFX, but upwards in VRML – you probably have to swap X or Z as well or the tiles will be mirrored …)

Share this post


Link to post
Share on other sites

I'll just invert Y for now. and will invert Z later if an asymmetrical model looks wrong. That's easier than trying to work it out properly. :)

Now, I just need to move on beyond the simplest terrain type...

Share this post


Link to post
Share on other sites

Well, I'm still cherry-picking which models to attempt, but making some small steps in the time I have.

 

Here's something asymmetrical, but I can't get the same orientation as 3View by inverting either X or Z. In fact if I do that, the Z-buffering seems to fail.

stone_1.png.b748f7a08e9e97850f0efdfae66ef97b.png

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×