Jump to content
COMBATSIM Forum
DrKevDog

Undocumented Ops

Recommended Posts

The work that was done on TFX3 .3 object models identified most working operation (opcodes) instructions completely as well as many with functions that remain unknown. Not much work has gone into identification and/or analysis of undocumented opcodes.

Mikew and Krycztij were able to uncover useful information related to game memory formatting, including Optimizations, memory format, format handling, etc.. That information I have incorporated into a method to analyze undocumented operation / instructions and map out their operand strings.

Considering that TFX3 was built upon TFX2 which was built upon TFX, it seemed reasonable to me that there should be some undocumented TFX3 instructions for even further future expansion.

One of my personal  interests in periodically rooting around in this classic legacy game is the area of color and transparency graphics. TFX3 makes it fun to play with. In TFX3 there was the initial introduction of non-indexed 24 bit color and 32 bit color. The implementation, however, was very limited.  I seem to be learning new things about color and transparency from some of  my recent discoveries.

These are some of the new instructions that I am currently working on:

00A0

00A1

00A4<offset>

00A6<offset>

00A5<offset>

00A9 terminator?

00AD <unknown> <unknown><unknown><unknown><unknown><unknown>

 

00A0 and 00A1 seem to be involved in shadow casting transparency /  Vertex Alpha?

00A4, 00A5, 00A6 and 00A9 seem to effect alpha transparency and blending.

Example: 00A4 the addition of 00A4 has an effect on Alpha Blend Mode and/or State, either single or stacked, increases the Alpha Transparency value (increased opacity) approximately 20% per occurrence.

This is effective on 32-bpp RGBA textures (with BGRA output images) and RGB using alpha images (00A7).

 

Without 00A4:

capture_003_25042017_171001.jpg.db3194bc3ec763820e97e994f78a926f.jpg

 

With 00A4 (4X stacked):

capture_001_25042017_165555.jpg.cc28c08395f0d2ff172572f1246a5766.jpg

Note the Blue / Red and Yellow / Cyan (a) swap due to the BGRA output of the 32 bit images. I suspect this is an TFX3 efficiency optimization which takes advantage of the way images are written in Little Endian memory format.

Share this post


Link to post
Share on other sites

Interesting. I don't remember ever trying to find a list of opcodes in the exe.

I haven't seen any of these extra opcodes in any of the 'noname' files, but that doesn't mean that they are not there.

Share this post


Link to post
Share on other sites

I have never seen a list of opcodes in the .exe. It would be helpful to have if such a list does exist.

I am working on opcode # 0006 which I have never seen in any file in TFX2 or 3. Currently it has me stumped. In memory the format suggests the following:

0006 <color> <pos0><pos1><pos2> <double word>

Any idea what that double word might be?

Share this post


Link to post
Share on other sites

This post indicates that I may have come across opcode 0006 at some point:

...it could have been a false positive, but it was so long ago, I can't remember.

The exe must parse the .3 opcode somehow, but I haven't found where in the disassembled code.

Share this post


Link to post
Share on other sites

Do you have any record of the existence in any of the object files, or not, of the following opcodes:

0010, 0011, 0012, 0013 or 0014 ?

 

Share this post


Link to post
Share on other sites

No, I don't have any recollection of any of those
Again, I have to say it was a long time ago since I looked for new opcodes, although I'm fairly sure that there is nothing new to discover in the TAW files.
We've barely scratched the surface of TFX1 though.

Share this post


Link to post
Share on other sites

I agree, and I suspect they are undocumented in all of the three TFX file bundles.

I am using my new technique on them.

- starting with 0011: Visibility test

// 0011 <position0> <position1> <offset>


If the line of the (transformed) vertices at the given positions faces away from the viewer, jump

<offset> - 2 bytes relative to the end of the instruction.

TFX3 has no Z-buffer, therefore this instruction could be used to assist with maintaining the

back-to-front draw order in the same, or similar way as the 0015 instruction. As I recall TFX

uses the 0015 instruction, exclusively, for the hyperplanes created for this purpose. Perhaps

there are specific instances in which a tri-poly-hyperplane is not the optimal solution (?).

Share this post


Link to post
Share on other sites

Two points don’t define a plane … how does 0011 decide to jump, then? Maybe distance related (point 1 closer to the viewer than point 2)?

Share this post


Link to post
Share on other sites

Good point ! While it might be possible that an line algorithm could be involved, that is not yet within my grasp of the concepts. It would make sense then to assume your distance theory and I will work to get some better control over the facts ^_^

Share this post


Link to post
Share on other sites

The facts, after additional testing, indicate that the jump executes if the camera position is closer to vertex <position0> than vertex <position1>.

Is this related to implementing perspective correct interpolation?

It's interesting that using the diagonal on a quad enables the jump with triggers both in the horizontal as well as vertical plane.

Share this post


Link to post
Share on other sites
On 5/9/2017 at 2:28 AM, DrKevDog said:

Is this related to implementing perspective correct interpolation?

Sounds like an optimized version of the triangle jump to me. Like „hey we can save two bytes and 20 CPU cycles if we use two points instead of three“; Art department saying „this is really hard to understand for our artists when they make the models“; everyone going back to the triangle version …

Share this post


Link to post
Share on other sites

And the last thing you want for your costly project is to piss the artists off, lest you risk ending up with art and models and graphics that look like this:

59151dd148204_mikewspic.jpg.5aeabb0d078f8a0b0b61e63abc9c5feb.jpg

Not that there's anything wrong with that...um...well maybe :P

 

On 5/5/2017 at 2:57 PM, Krycztij said:

Two points don’t define a plane … how does 0011 decide to jump, then? Maybe distance related (point 1 closer to the viewer than point 2)?

The difficulty I am having with accepting the 2 point version of the triangle jump explanation for this instruction is that the triangle jump uses a 3D-Hyperplane for 3D-models but a 2D-hyperplane can only be used on 2D-models. In TFX/2/3/ the 3D-hyperplanes are used exclusively for obtaining a back-to-front draw order but that is simply not necessary for flat 2D models and that makes the presence of such an instruction, for that purpose: illogical to me. I can think of all sorts of fun ways that I could possibly use that opcode, however, I hope to find the definitive purpose for which they included that instruction :angry:

 

Share this post


Link to post
Share on other sites
3 hours ago, DrKevDog said:

that makes the presence of such an instruction, for that purpose: illogical to me.

Could this be the reason it’s unused? :)

Considering what has been said about EF2000 (“project from hell”, “a million lines of pants-down code”), occurance of such opcodes doesn’t surprise me at all.

Share this post


Link to post
Share on other sites

Time for some rampant speculation...

This is roughly the timeline for DiD's games:
F29 Retaliator (1989 Amiga, 1991 PC)
EPIC (1992 Amiga first, then PC)
Robocop 3 (1992 Amiga & PC)
TFX (1993 PC First, then Amiga)
Inferno (1994 PC only)
EF2000 (1995-1997 various releases)
F-22: ADF (1997)
F-22 TAW (1998)

We know that .3 files, were used from TFX onwards, with increasingly complex opcodes. While similar functions are grouped together, the numbering seems quite arbitrary.

It is not inconceivable that the basic system (with DKD's opcodes) can be traced all the way to back to F29 Retaliator and that the .3 parser is Amiga code that has been cross-compiled to x86 somehow. This may explain why IDA doesn't seem to do a particularly good job of dissassembling where I think the .3 file handling should be.

Share this post


Link to post
Share on other sites

Here's a comparision bewteen EF2000's cloud1.3 and an unidentified file from the Robocop3 dat file. The first 36 or so bytes are in a similar, so it seems to be a .3 file. The rest of it looks completely different. Not sure what this proves though...
robovsef.png.10bbaab8ea81e98a6466c24e9dd8d8cc.png

Share this post


Link to post
Share on other sites

The header format looks identical. Eluding to the comment you made about the increasing complexity of the opcodes as TFX evolved, I wonder if the Robocop executable would make for easier work finding where the .3 handling could be located and give some clues on how to find it in the later programs?

Share this post


Link to post
Share on other sites
On 5/12/2017 at 1:25 AM, Krycztij said:

Could this be the reason it’s unused? :)

LOL! I have thought of that at least a hundred times. The idea of it simply being useless code is, IMHO, not only unpleasant but, more disturbing, currently unprovable :mellow:

Share this post


Link to post
Share on other sites
8 hours ago, DrKevDog said:

... I wonder if the Robocop executable would make for easier work finding where the .3 handling could be located and give some clues on how to find it in the later programs?

Possibly. Your use of the word 'easier' may be tecnnically correct, but that's a path I don't intend to take. :)

I think that's enough for Robocop3. Of archeological interest, the datafile starts with a 4 byte address, to the 'list section', then all the files which are mostly RA compressed. The list section contains the paths and extensions, but the list only contains 8 bytes for each file containing the size and name hash. The files are placed in the same order as the list so the offset to each file can be deduced from the size.

The paths and extensions are as follows:

corridor\robo\
corridor\men\
corridor\
robo.voc\
iff\256\
colours\
anim\
dat\
smb\
spr\
ig\
3\

.dat
.drv
.fgt
.bin
.256
.col
.cor
.shl
.hdd
.hud
.3dl
.lbm
.voc
.smb
.spr
.ig
.3

Of note, is the lack of ssd.

Share this post


Link to post
Share on other sites

So, digging deeper with EPIC.

While in Robocop3, it looks like we could extract and decompress all the files but we don't know their names, the opposite seems to be true for EPIC.

The datafile doesn't contain a list, but the filenames, paths and extensions can be deduced from the game exe. The sad thing is that the files are compressed in JR format instead of RA. It is likely the schemes aren't that different but it would take further effort.

The obvious paths and extensions, there may be others:

3\
anim\
dat\
iff\256\
ig\
levs\
voc\

.3
.256
.mp
.smb
.hdd
.hud
.bin
.3dl
.lbm
.ig
.dat
.voc

Share this post


Link to post
Share on other sites

I am following your archeological ventures with good interest. One of my concerns with the undocumented opcodes is that, despite reasonable efforts to define their function, without a formal game example / model, the definitions remain unprovable and mere speculation.

 

 

On 5/12/2017 at 2:14 AM, mikew said:

 

We know that .3 files, were used from TFX onwards, with increasingly complex opcodes. While similar functions are grouped together, the numbering seems quite arbitrary.

 

That similar functions are grouped together is helpful. I'm currently working on undocumented opcodes 0028 and 0029. Like 0026 and 0027, we can assume some similar function.

0027 <threshold> <offset>

If the object’s distance from the viewer is above or equals <threshold>, jump <offset> - 2 bytes.

Used to fade details, e.g. shadows, with distance.

This is what I have discovered for 0028:

format: 0028 <unknown><unknown>

 

Many of the values I plugged in caused a program fault but a few caused a division overflow. That led to an assumption that, because a division overflow is not very common, I could work with divisional increments, so I began playing around with some integers and found a pair that resulted in the following:

capture_001_16052017_201258.jpg.58f5cf185d128147d6b4cd8c21b3981b.jpg

capture_002_17052017_192115.jpg.33f2f918eec2740166ed2dc61d4ee7cd.jpg

Believe it or not, that shadow is cast from the windsock (awind.3) by simply adding 002800C51770 to the front of the bytecode. It seems 0028 handles horizontal shadow casting while 0029 handles vertical shadows. I still get some instability and the file modified will cast shadows off of any nearby object at times and other times is stable. The values I used for the unknowns were an educated guess and I am still trying to know what those unknowns really are.

Share this post


Link to post
Share on other sites

Amazing!

I've come to a bit of a dead-end with F29-Retaliator. I can't see any pattern in the data files at all. :(

Share this post


Link to post
Share on other sites

100 Bitcoins deserves the man who persevered to analyze the timeline all the way back to F29-Retaliator. My hats off to you :thumbsup:

I'm still working on opcodes 0028 and 0029, hopefully some progress by next post.

Share this post


Link to post
Share on other sites

Well, I have a bit of time on my hands this week, but unfortunately can only do basic datafile analysis on the machine I have available.

If I can understand the JR compression format, it will be possible to get at EPIC's 3 files. There is some hope as the scheme doesn't seem to pack the file as tightly as RA. The following picture shows a RA compressed .LBM, together with its corresponding decompressed file. The third is (presumably) a JRd .LBM file from EPIC.
ra_vs_jr.png.c7202f52e922dab220c0f2f68ebdee05.png

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×