DrKevDog

Undocumented Ops

43 posts in this topic

Oh, this is even more interesting when you consider that ILBMs are already compressed! In order to not make the resulting file smaller at all, the ILBM needs to be stored in an uncompressed format before JR’ing it.

I’ve seen some RLE compressions on Atari and PlayStation, and I can only recommend:

  1. pick a human-readable file like TXT or INI to begin with
  2. find consecutive chunks (e.g. the 12 B of “FORM----ILBM”, then 2B of “HD”)
  3. list them with the leading codes (in this case “02 00 F2 BA 00 00 63 B8” and “01”)
  4. unless you are very gifted at cracking codes, translate the leading codes to binary (these algorithms are always bit-packed and it’s very hard to find patterns in the hex numbers)
  5. try finding patterns, of course
  6. sort the consecutive chunks by length or by frequency of occurancy, maybe this reveals identical bits
  7. consider that these algorithms may be using words instead of bytes

I guess it’s something RLE-related, because “01” just before the two “HD” bytes seems too much of a coincidence. Maybe you get a clue from PackBits, the ILBM RLE compression:

auto n = next_byte();
if(n <= 127) {
	// copy the following n + 1 bytes literally
} else if(n > 128) {
	// replicate the following byte 257 - n times (the numbers make much more sense if you consider the byte signed, but I didn’t for performance, sorry!)
} else {
	// reserved; ignore
}

… or the PCX RLE compression:

auto n = next_byte();
if(0xC0 <= n) {
	auto const count = 0x3F & n;
	// replicate the following byte this many times
} else {
	// copy the byte
}

… or something from PlayStation:

auto n = next_byte_signed();
if(n < 0) {
	// repeat the following byte 2 + |n| times
} else {
	// copy the following 1 + n bytes
}

The pattern is always the same, just the magic numbers differ a lot (they’re often hand-tuned for good results with the game).

Share this post


Link to post
Share on other sites

Thanks!

The 02 00 is part of the header and F2 BA probably the size of the uncompressed file like in RA. It looks like the first 12 bytes FORM... aren't compressed at all.

I didn't show enough of the RAd LBM file above. If we look a bit further, we see the FORM etc.
ra.png.4e55f188c670c673c3792f3403925eb9.png

I was hoping that RA and JR might be basically the same thing, but with a different magic number like the name hash routine, but unfortunately not.

Share this post


Link to post
Share on other sites

RA is Huffman, and as such it has very high CPU demand (lots of ALU and cache pressure). I have not seen anything like that in early 1990ies systems (even sliding windows were rare), so I stand by my opinion – it is very likely RLE compression :)

Share this post


Link to post
Share on other sites

The 0028 opcode is more interesting than at first. It has me rooting around

in the executable and wishing there was more known about the structure and

format of the executable.

format: 0028 <unknown><unknown>

Unknown #1 appears to set the distance at which action is initiated

Unknown #2 is the offset to the routine in bytes (-2B)

 

The unusual thing about the unknown #2 in the 0028 instruction is that it

seems to disregard .3 file headers and footers and treats the executable

like a container. It can execute the shadows routine of one .3 file when

called from another, simply by adding a proper 0028 line to the recipient

file. An array of 0028's can do some interesting things.

 

Below is an image of Hangl_90.3 with its file-coded shadow in the back but

also with the shadow contained in the Hangs90.3 file-code in the front.

hangL90-hangs90-04062017.jpg.72c25d80b3ba2d9fcb5eff4ed7e08138.jpg

It's like I have now acquired the ability to bend light, bi-directionally :lol:

Share this post


Link to post
Share on other sites

SPECULATION ALERT!

 

I am speculating that 0028 is a different type of instruction/opcode than we

have been studying before, which is perhaps why it is not found in any of  the game files. It seems to be a practical opcode for use during the development phase.  0028 can render a wide range of routines and subroutines. It would be useful to the graphics artists who were working on the 3d models.  Let’s say, for example, an artist is working on the visuals for the damage  states of the Luxor tile, and lets say they are working on  the lux_radr.3  model and would like to visualize the various LOD and damage state models in-game. They could use 0028 to simultaneously render the various damage state models, in-game, for

inspection and potential modifications.

Simply place the 0028, or an array of such, in the lux_radr.3 file and configure it so that the second operand word calls the specific damage model subroutine and the first operand word 

defines the distances at which the called model will be rendered. The specific model can be further manipulated, for analysis, by placing modification codes, such as 0030 (scale) ahead of the 0028 instruction.

0028002F15EC

In this example I have set the distance so that only the undamaged model is visible beyond distance 002F.

From distance 0 to 2F I also call the subroutine for damage state 1:

capture_004_24062017_152258.jpg.9f8a40a7ba3fc0ba718e73ac1a36b7b5.jpg

 

003000020028002F15EC

This example is identical to the above, except I scaled the damage model up to unmask it completely.

capture_003_24062017_112251.jpg.57f27b16dbe18386a4e11fe571fe7fcd.jpg

 

Share this post


Link to post
Share on other sites

Inspired by your selfless dedication, I've taken another look at EPIC.dat.
While the files may be RLE compressed, I don't see a simple pattern.

EPIC.dat contains 230 separate JR files, but the middle two thirds or so of the file looks to be filler...but of course it may an important part of the compression process that I'm not seeing.

After the JR, a '0200' and 4 bytes which look like the length of the uncompressed file, we have two mysterious bytes which are linked in some way to the payload.
Here are some examples of those two bytes together with the following 48 bytes.

These seem to be .3 files with up to at least the 7fff untouched. All these files with the same leading two bytes have the 7fff in the same position.
7c5c fb500fa01b581e141e781edc1f407fff000200028039000301000e1ff023b2250e104c25c801006300642161130764ff
7c5c fb500fa01b581e141e781edc1f407fff000200028039000301000e2e4a320c330e10a6342201007a0064213113c805ff
0f17 f510119423282ee032c836b07fff0002400e0002000301000e213804e2351c35fe010d4c3100fda801f4fb50025803e8
0f17 f51011941b582ee032c836b07fff0002400e0002000301000e123804d21fb02092010d2e3100fda801f4fb50025803e8
3c5c fd1211942134222e23287fff000200022039000301000e128a402113ce010d6801900121182cfdb20500aafd440571a0
3c5c ffd8012c01f4025802bc7fff000200022039000301000e00420129005a010004fffe26c705282105dcd04111052f6882
1e00 f5101b58bc00a5557fff000200b01c02000301000e336042fa010da0fda8018801f4fb50025803e83bfc181b6c050d11
1e00 ff5109c42bc03a557fff000200b01c02000301000e0560423a010d5000afff941bf40019050305c121ffe711055dffdb

All .lbm files have 63b8 at the start. The first 30 or so bytes are mostly uncompressed and we can see the 320x200 (0x140,0xc8)
63b8 464f524d0000d172494c424d014844c0010b0014014000c8070008000101c2041005060f434d415000900c0316cfefff
63b8 464f524d0000d042494c424d014844c0010b0014014000c8070008000100e10505060f434d41500fc80b0316d300a313
63b8 464f524d0000d03a494c424d014844c0010b0014014000c8070008000101c204fd05060f434d41505c900c0316cf0073
63b8 464f524d0000ba0a494c424d014844c0010b0014014000c8070008000100e10505060f434d415033c80b0316ff02fba2
63b8 464f524d00008b80494c424d014844c0010b0014014000c8070008000100e10505060f434d415033c80b0316ff02f733

These look fine in Notepad, but the 'clever' forum SW screws up the format as usual...

Anyway, I'm seeing a similar .3 header format to the following games, with a negative 16-bit number, followed by mostly rational LOD distances, then the 7fff.

Share this post


Link to post
Share on other sites
4 hours ago, mikew said:

Anyway, I'm seeing a similar .3 header format to the following games, with a negative 16-bit number, followed by mostly rational LOD distances, then the 7fff.

Speaking of .3 headers, IIRC, not until tfx3 did they introduce the upgrade-transition from the class 00 shape files to 83. The 83 files are much easier to work with so I am trying to find an easy conversion process. It seems that the main factors are the presence and number of LOD's, and the presence or absence of a header texture and its length, don't recall if render order is a factor also, but didn't you work this all out at some time in the past?

Share this post


Link to post
Share on other sites
2 hours ago, DrKevDog said:

It seems that the main factors are the presence and number of LOD's, and the presence or absence of a header texture and its length, don't recall if render order is a factor also, but didn't you work this all out at some time in the past?

I’m re-constructing this from code comments, so it will probably not be 100 % right, but:

  1. My code handles hard-coded textures names in any class (00, 04, 20, 83, 87, A3). I can’t recall if it was out of necessity or just for simplifying code.
  2. TFXplorer uses rendering priority directly, not taking into account shape classes. However, this only applies to terrain shapes because planes/tanks/… are handled entirely different from TAW’s rendering.

Summary: I don’t know anything for sure :(

Share this post


Link to post
Share on other sites

I think 83,87 and A3 are only associated with F22ADF and TAW.
The interesting thing is that some characteristics of the later .3 files can be seen in EPIC. It's a pity I don't understand JR compression so that we could look for your missing opcodes. It looks like the TAW code still looks for both JR and RA so if I cut out some of the .3 files from Epic.dat and transplanted them into TAW's did.dat, there's a fair chance that TAW would try and render them. I don't really have time for that though. :(

Share this post


Link to post
Share on other sites
On 6/25/2017 at 9:49 PM, mikew said:

I think 83,87 and A3 are only associated with F22ADF and TAW.

My list:

  • 00 TFX2 entity
  • 04 TFX2 sub-entity
  • 20 TFX2/3 terrain without LOD
  • 83 TFX3 entity
  • 87 TFX3 sub-entity
  • A3 TFX3 terrain with LOD

With “sub-entity” meaning that it is typically attached to something else, e.g. many weapon shapes are sub-entities to planes.

On 6/25/2017 at 9:49 PM, mikew said:

It's a pity I don't understand JR compression so that we could look for your missing opcodes. It looks like the TAW code still looks for both JR and RA so if I cut out some of the .3 files from Epic.dat and transplanted them into TAW's did.dat, there's a fair chance that TAW would try and render them. I don't really have time for that though. :(

Wow, that’s an interesting find! If I just had some time, I’d look at JR myself …

Share this post


Link to post
Share on other sites

       

Time is a problem...
For information only, we know that 'JR' in Intel format is 21066 and 'RA'= 16722 and we see subroutines like this in TAW.

 


void __usercall sub_5974E0(int a1<ebx>, int a2<ebp>)
{
  int v2; // eax@3
  __int16 v3; // dx@4
  int v4; // ecx@4
  int v5; // esi@4

  if ( a1 )
  {
    v2 = sub_596E60();
    if ( v2 )
    {
      v5 = *(_DWORD *)(v2 + 8);
      sub_596FE4();
      v3 = *(_WORD *)(a1 + 560);
      if ( v3 == 21066 )
      {
        sub_526D3C(*(_DWORD *)a1);
        sub_5F8778(a2);
      }
      else
      {
        if ( v3 == 16722 )
        {
          sub_5F8428(v5, v4, *(_DWORD *)a1);
        }
        else
        {
          bzero_();
          sub_526D3C(*(_DWORD *)a1);
        }
      }
    }
  }
}


..where for RA sub 5f8428 is:

 


void __usercall sub_5F8428(int a1<esi>, int a2, int a3)
{
  signed int v3; // eax@1
  int v4; // edx@1
  int v5; // edi@1
  unsigned __int8 v6; // cf@3
  int v7; // esi@5
  char *v8; // ebx@7
  int v9; // edx@8
  int v10; // ecx@8
  char v11; // al@11
  int v12; // edx@11
  int v13; // ecx@12
  unsigned __int8 v14; // cf@15
  char v15; // al@5
  unsigned __int8 v16; // tt@5
  unsigned __int8 v17; // cf@5
  char v18; // al@5
  char v19; // ah@10
  int v20; // ecx@11
  char v21; // ah@13
  char v23; // al@18
  unsigned __int8 v24; // tt@18
  unsigned __int8 v25; // cf@18
  char v26; // al@18

  dword_72BF1C = a3;
  dword_72C124 = 12;
  v5 = a2;
  sub_5F8735();
  sub_5F8735();
  LOWORD(dword_72D6E0) = sub_5F8735();
  HIWORD(dword_72D6E0) = sub_5F8735();
  word_72D6E8 = sub_5F8735();
  word_72D6EA = sub_5F8735();
  v3 = 512;
  if ( (unsigned int)dword_72D6E0 <= 0x200 )
    v3 = dword_72D6E0;
  dword_72C124 = v3;
  dword_72D6E4 = v5;
  v6 = __MKCADD__(-128, -128);
  if ( v4 - 1 < 0 )
    sub_5F8750();
  v15 = *(_BYTE *)a1;
  v7 = a1 + 1;
  v16 = v6;
  v17 = __MKCADD__(v6, v15);
  v18 = v16 + v15;
  if ( v17 | __MKCADD__(v18, v18) )
  {
    do
    {
      dword_72BF20 = (unsigned __int16)sub_5F8735();
      sub_5F8665();
      sub_5F8665();
      v8 = &byte_72DF10;
      sub_5F8665();
      do
      {
        sub_5F85DD();
        for ( ; v10; --v10 )
        {
          --v9;
          if ( v9 < 0 )
            sub_5F8750();
          v19 = *(_BYTE *)v7++;
          *(_BYTE *)v5++ = v19;
        }
        v11 = sub_5F856E();
        if ( v20 )
        {
          v11 = sub_5F860E();
          v8 += v5;
          do
          {
            v21 = *v8++;
            *(_BYTE *)v5++ = v21;
            --v13;
          }
          while ( v13 );
        }
      }
      while ( dword_72BF20-- != 1 );
      v14 = __MKCADD__(v11, v11);
      if ( !(2 * v11) )
      {
        if ( v12 - 1 < 0 )
          sub_5F8750();
        v23 = *(_BYTE *)v7++;
        v24 = v14;
        v25 = __MKCADD__(v14, v23);
        v26 = v24 + v23;
        v14 = v25 | __MKCADD__(v26, v26);
      }
    }
    while ( v14 );
    if ( word_72D6EA != (unsigned __int16)sub_5F86FC() )
      __asm { int     3               ; Trap to Debugger }
  }
  else
  {
    __asm { int     3               ; Trap to Debugger }
  }
}


..and for JR sub 5f8778 is:

 


__int16 __usercall sub_5F8778<ax>(int a1<ebp>)
{
  unsigned __int8 v1; // cf@1
  unsigned int v2; // ecx@1
  int v3; // ebx@1
  void *v4; // edi@1
  int v5; // esi@1
  unsigned __int8 v6; // cf@2
  __int16 v7; // ax@2
  unsigned __int8 v8; // cf@3
  __int16 result; // ax@3
  unsigned __int8 v10; // cf@4
  unsigned __int8 v11; // cf@5
  unsigned __int8 v12; // cf@6
  __int16 v13; // ax@14
  unsigned __int8 v14; // cf@22
  __int16 v15; // ax@22
  unsigned __int8 v16; // cf@24
  unsigned __int8 v17; // cf@29
  char v18; // zf@35
  unsigned __int8 v19; // cf@38
  unsigned __int8 v20; // cf@40
  unsigned __int8 v21; // cf@42
  unsigned __int8 v22; // cf@46
  __int16 v23; // ax@46
  unsigned __int8 v24; // cf@48
  unsigned __int8 v25; // cf@54
  __int16 v26; // ax@2
  __int16 v27; // tt@2
  unsigned __int8 v28; // cf@2
  __int16 v29; // ax@2
  __int16 v30; // ax@3
  __int16 v31; // tt@3
  unsigned __int8 v32; // cf@3
  __int16 v33; // ax@3
  __int16 v34; // ax@4
  __int16 v35; // tt@4
  unsigned __int8 v36; // cf@4
  __int16 v37; // ax@4
  __int16 v38; // ax@5
  __int16 v39; // tt@5
  unsigned __int8 v40; // cf@5
  __int16 v41; // ax@5
  __int16 v42; // ax@6
  __int16 v43; // tt@6
  unsigned __int8 v44; // cf@6
  __int16 v45; // ax@6
  char v46; // zf@8
  __int16 v47; // ax@14
  __int16 v48; // tt@14
  unsigned __int8 v49; // cf@14
  __int16 v50; // ax@14
  char v51; // cf@14
  char v52; // zf@15
  char v53; // zf@16
  char v54; // zf@18
  char v55; // zf@20
  char v56; // zf@22
  __int16 v57; // ax@23
  __int16 v58; // tt@23
  unsigned __int8 v59; // cf@23
  __int16 v60; // ax@23
  char v61; // zf@24
  __int16 v62; // ax@25
  __int16 v63; // tt@25
  unsigned __int8 v64; // cf@25
  __int16 v65; // ax@25
  char v66; // zf@29
  __int16 v67; // ax@30
  __int16 v68; // tt@30
  unsigned __int8 v69; // cf@30
  __int16 v70; // ax@30
  __int16 v71; // ax@33
  __int16 v72; // tt@33
  unsigned __int8 v73; // cf@33
  __int16 v74; // ax@33
  char v75; // cf@33
  char v76; // zf@38
  char v77; // zf@40
  char v78; // zf@42
  char v79; // zf@46
  __int16 v80; // ax@47
  __int16 v81; // tt@47
  unsigned __int8 v82; // cf@47
  __int16 v83; // ax@47
  char v84; // zf@48
  __int16 v85; // ax@49
  __int16 v86; // tt@49
  unsigned __int8 v87; // cf@49
  __int16 v88; // ax@49
  __int16 v89; // ax@52
  __int16 v90; // tt@52
  unsigned __int8 v91; // cf@52
  __int16 v92; // ax@52
  __int16 v93; // ax@53
  __int16 v94; // tt@53
  unsigned __int8 v95; // cf@53
  __int16 v96; // ax@53
  char v97; // zf@54
  __int16 v98; // ax@55
  __int16 v99; // tt@55
  unsigned __int8 v100; // cf@55
  __int16 v101; // ax@55

  v4 = *(void **)(a1 + 12);
  v5 = *(_DWORD *)(a1 + 8) + 8;
  HIWORD(v3) = 0;
  HIWORD(v2) = 0;
  v1 = 1;
LABEL_33:
  v71 = *(_WORD *)v5;
  v5 += 2;
  v72 = v1;
  v73 = __MKCADD__(v1, v71);
  v74 = v72 + v71;
  v75 = v73 | __MKCADD__(v74, v74);
  result = 2 * v74;
  if ( !v75 )
  {
LABEL_34:
    *(_BYTE *)v4 = *(_BYTE *)v5++;
    v4 = (char *)v4 + 1;
    goto LABEL_35;
  }
  while ( 1 )
  {
    LOWORD(v2) = 2;
    BYTE1(v3) = 0;
    v19 = __MKCADD__(result, result);
    v76 = 2 * result == 0;
    v13 = 2 * result;
    if ( v76 )
    {
      v47 = *(_WORD *)v5;
      v5 += 2;
      v48 = v19;
      v49 = __MKCADD__(v19, v47);
      v50 = v48 + v47;
      v51 = v49 | __MKCADD__(v50, v50);
      v13 = 2 * v50;
      if ( !v51 )
        goto LABEL_15;
    }
    else
    {
      if ( !v19 )
      {
LABEL_15:
        v6 = __MKCADD__(v13, v13);
        v52 = 2 * v13 == 0;
        v7 = 2 * v13;
        if ( v52 )
        {
          v26 = *(_WORD *)v5;
          v5 += 2;
          v27 = v6;
          v28 = __MKCADD__(v6, v26);
          v29 = v27 + v26;
          v6 = v28 | __MKCADD__(v29, v29);
          v7 = 2 * v29;
        }
        LOBYTE(v2) = 2 * (v6 + 2);
        v8 = __MKCADD__(v7, v7);
        v53 = 2 * v7 == 0;
        result = 2 * v7;
        if ( v53 )
        {
          v30 = *(_WORD *)v5;
          v5 += 2;
          v31 = v8;
          v32 = __MKCADD__(v8, v30);
          v33 = v31 + v30;
          v8 = v32 | __MKCADD__(v33, v33);
          result = 2 * v33;
        }
        if ( v8 )
        {
          v10 = __MKCADD__(result, result);
          v54 = 2 * result == 0;
          result *= 2;
          if ( v54 )
          {
            v34 = *(_WORD *)v5;
            v5 += 2;
            v35 = v10;
            v36 = __MKCADD__(v10, v34);
            v37 = v35 + v34;
            v10 = v36 | __MKCADD__(v37, v37);
            result = 2 * v37;
          }
          LOWORD(v2) = (_WORD)v2 - 1;
          LOBYTE(v2) = 2 * (v10 + (_BYTE)v2);
          if ( (_BYTE)v2 == 9 )
          {
            LOBYTE(v2) = (_BYTE)v2 >> 1;
            do
            {
              v12 = __MKCADD__(result, result);
              v46 = 2 * result == 0;
              result *= 2;
              if ( v46 )
              {
                v42 = *(_WORD *)v5;
                v5 += 2;
                v43 = v12;
                v44 = __MKCADD__(v12, v42);
                v45 = v43 + v42;
                v12 = v44 | __MKCADD__(v45, v45);
                result = 2 * v45;
              }
              BYTE1(v3) = 2 * (v12 + BYTE1(v3));
              --v2;
            }
            while ( v2 );
            LOBYTE(v2) = 2 * (BYTE1(v3) + 3);
            while ( v2 )
            {
              *(_WORD *)v4 = *(_WORD *)v5;
              v5 += 2;
              v4 = (char *)v4 + 2;
              --v2;
            }
            goto LABEL_35;
          }
        }
        goto LABEL_20;
      }
    }
    v20 = __MKCADD__(v13, v13);
    v77 = 2 * v13 == 0;
    result = 2 * v13;
    if ( v77 )
    {
      v89 = *(_WORD *)v5;
      v5 += 2;
      v90 = v20;
      v91 = __MKCADD__(v20, v89);
      v92 = v90 + v89;
      v20 = v91 | __MKCADD__(v92, v92);
      result = 2 * v92;
    }
    if ( !v20 )
      goto LABEL_32;
    LOWORD(v2) = 3;
    v21 = __MKCADD__(result, result);
    v78 = 2 * result == 0;
    result *= 2;
    if ( v78 )
    {
      v93 = *(_WORD *)v5;
      v5 += 2;
      v94 = v21;
      v95 = __MKCADD__(v21, v93);
      v96 = v94 + v93;
      v21 = v95 | __MKCADD__(v96, v96);
      result = 2 * v96;
    }
    if ( v21 )
      break;
LABEL_20:
    v11 = __MKCADD__(result, result);
    v55 = 2 * result == 0;
    result *= 2;
    if ( v55 )
    {
      v38 = *(_WORD *)v5;
      v5 += 2;
      v39 = v11;
      v40 = __MKCADD__(v11, v38);
      v41 = v39 + v38;
      v11 = v40 | __MKCADD__(v41, v41);
      result = 2 * v41;
    }
    if ( !v11 )
      goto LABEL_32;
    v14 = __MKCADD__(result, result);
    v56 = 2 * result == 0;
    v15 = 2 * result;
    if ( v56 )
    {
      v57 = *(_WORD *)v5;
      v5 += 2;
      v58 = v14;
      v59 = __MKCADD__(v14, v57);
      v60 = v58 + v57;
      v14 = v59 | __MKCADD__(v60, v60);
      v15 = 2 * v60;
    }
    BYTE1(v3) = 2 * v14;
    v16 = __MKCADD__(v15, v15);
    v61 = 2 * v15 == 0;
    result = 2 * v15;
    if ( v61 )
    {
      v62 = *(_WORD *)v5;
      v5 += 2;
      v63 = v16;
      v64 = __MKCADD__(v16, v62);
      v65 = v63 + v62;
      v16 = v64 | __MKCADD__(v65, v65);
      result = 2 * v65;
    }
    if ( v16 )
    {
      v22 = __MKCADD__(result, result);
      v79 = 2 * result == 0;
      v23 = 2 * result;
      if ( v79 )
      {
        v80 = *(_WORD *)v5;
        v5 += 2;
        v81 = v22;
        v82 = __MKCADD__(v22, v80);
        v83 = v81 + v80;
        v22 = v82 | __MKCADD__(v83, v83);
        v23 = 2 * v83;
      }
      BYTE1(v3) = (unsigned __int8)(2 * (v22 + BYTE1(v3))) | 4;
      v24 = __MKCADD__(v23, v23);
      v84 = 2 * v23 == 0;
      result = 2 * v23;
      if ( v84 )
      {
        v85 = *(_WORD *)v5;
        v5 += 2;
        v86 = v24;
        v87 = __MKCADD__(v24, v85);
        v88 = v86 + v85;
        v24 = v87 | __MKCADD__(v88, v88);
        result = 2 * v88;
      }
      if ( v24 )
        goto LABEL_32;
      goto LABEL_29;
    }
    if ( !BYTE1(v3) )
    {
      ++BYTE1(v3);
LABEL_29:
      v17 = __MKCADD__(result, result);
      v66 = 2 * result == 0;
      result *= 2;
      if ( v66 )
      {
        v67 = *(_WORD *)v5;
        v5 += 2;
        v68 = v17;
        v69 = __MKCADD__(v17, v67);
        v70 = v68 + v67;
        v17 = v69 | __MKCADD__(v70, v70);
        result = 2 * v70;
      }
      BYTE1(v3) = 2 * (v17 + BYTE1(v3));
    }
LABEL_32:
    LOBYTE(v3) = *(_BYTE *)v5;
    memcpy(v4, v4 + -v3 - 1, v2);
    v4 = (char *)v4 + v2;
    HIWORD(v2) = 0;
    ++v5;
LABEL_35:
    v1 = __MKCADD__(result, result);
    v18 = 2 * result == 0;
    result *= 2;
    if ( !v1 )
    {
      *(_BYTE *)v4 = *(_BYTE *)v5++;
      v4 = (char *)v4 + 1;
      v1 = __MKCADD__(result, result);
      v18 = 2 * result == 0;
      result *= 2;
      if ( !v1 )
        goto LABEL_34;
    }
    if ( v18 )
      goto LABEL_33;
  }
  LOBYTE(v2) = *(_BYTE *)v5++;
  if ( (_BYTE)v2 )
  {
    LOWORD(v2) = (_WORD)v2 + 8;
    goto LABEL_20;
  }
  v25 = __MKCADD__(result, result);
  v97 = 2 * result == 0;
  result *= 2;
  if ( v97 )
  {
    v98 = *(_WORD *)v5;
    v5 += 2;
    v99 = v25;
    v100 = __MKCADD__(v25, v98);
    v101 = v99 + v98;
    v25 = v100 | __MKCADD__(v101, v101);
    result = 2 * v101;
  }
  if ( v25 )
    goto LABEL_35;
  return result;
}

I have no idea whether these have anything to do with decompression though...and if it is, it's impossible to decode :(

Share this post


Link to post
Share on other sites

This forum software sucks big time...

I tried to write outside the spoiler tags:
I have no idea whether these have anything to do with decompression though...and if it is, it's impossible to decode :(

 

Share this post


Link to post
Share on other sites

Update on undocumented opcodes:

I seriously over-analyzed 0028 and 0029, they are the same as codes 0026 / 0027 except with a call instead of a jump.<_<

I was able to get a better analysis when i substituted them in ref_2.3 for 0043. Code 0043 is a Time of Day opcode and I seem to have discovered two additional, undocumented T.O.D. codes: 0044 and 0046. This analysis is greatly facilitated by the TAW Diagnostics, which has a feature for advancing time skip hour-by-hour or minute-by-minute. The previous work on T.O.D. codes lays out some of the bit mask data which is very helpful, thanks. Using the time skip diagnostics seems to provide some empirical data to suggest there may be a number of different bitwise operations used in this group of opcodes. if there have been any updates on the 003f thru 0046 operations, especially regarding less commonly used operations like "bit flipping", please post.

Share this post


Link to post
Share on other sites

Wow. One thing that annoys me a lot is that the time of day opcodes actually differ in between EF2000 and TAW. Whenever there are drawing errors in TFXplorer, it’s due to TOD opcodes having a different meaning in EF2000. E.g. there once was a frame rate slowdown at 12:00, and when I investigated, I found out that everything is drawn twice due to the TOD opcodes which I only analyzed for TAW.

I didn’t write down which specific shapes or opcodes were affected, though.

My comments on 003f…0046:

// 003F <time> <offset>
// If (1 << time of day) equals <time>, draw the submesh at <offset> - 2 B.
// Enables building lights and street lights at 22:00–06:00.

// 0040 <time> <offset>
// 0042 <time> <offset>
// If a bitwise AND of (1 << time of day) and <time> evaluates nonzero, draw the submesh at <offset> - 2 B.
// Enables building lights at 20:00–08:00. 0042 enables high-quality glare in TFX3’s “air_hq.3” and “oilrig.3”.

// 0043 <time> <offset>
// 0045 <time> <offset>
// If (2 << time of day) - 1 is above or equals <time>, jump <offset> - 2 B.
// Selects shadows and enables landing gear lights on planes at night.

 

Share this post


Link to post
Share on other sites

Interesting comment on the EF2000 / TAW difference. The tests I did on 0043 in ref_2.3 gave me double shadows. Ref_2 .3 might not be a good test model because it handles the shadows at 0043007F separate from all the others, for some reason. I'm going to retest and redo my calculations and post anything of significance.

Share this post


Link to post
Share on other sites
On 7/22/2017 at 5:25 PM, Krycztij said:

Wow. One thing that annoys me a lot is that the time of day opcodes actually differ in between EF2000 and TAW. Whenever there are drawing errors in TFXplorer, it’s due to TOD opcodes having a different meaning in EF2000.

 

Wow again! Much different.

Here is my update:

X

TOD

Mode

0043

VALUE

bit fields

 

 

 

 

 

 

 

 

 

 

0

00:00

NO  MOON  AT  NIGHT

0000

00000000 00000000

 

1

08:00

RED0800

0001

00000000 00000001

2

10:00

RED1000

0003

00000000 00000011

3

12:00

RED1200

0007

00000000 00000111

4

14:00

RED1400

000F

00000000 00001111

5

16:00

RED1600

001F

00000000 00011111

6

18:00

RED1800

003F

00000000 00111111

7

20:00

RED2000

007F

00000000 01111111

8

22:00

MOONLIT  NIGHT

00FF

00000000 11111111

9

06:00

RED0600

01FF

00000001 11111111

 Tfx has evolved the T.O.D. system and shadow implementation over its successive releases.

EXAMPLE:

Shadow.3 (tfx2)

FFF30BB8000000007FFF000000000000EA840000
0002000300030000000000000000000E034603460346000000000000001D
0030000A
00620000FF79FFCE0087
0066FF10
00650032
006600F0
0064010E
0065FFCE
0066FF10
00650032
0064FF19
00660000
00640060
00660000
00640060
00660000
0065FFC8
0064FFA0
00660000
0064FFA0
0067FFF1FFF2
006600F0
006400DE
0066FF10
0067FF4FFFF6
006600F0
00640084
0066FF10
0063FFDC005000F0
0068FF8EFF5B
006400A8
0061001D0000
001B000B000F001B00030004001C
001B000B000F001B000900020003
001B000B000F001C00040007000D
001A000B000F000800010002
001B000B000F0008001100120001
001B000B000F0012001100100016
001B000B000F000F001900160010
001B000B000F000F000E00150019
001B000B000F0015000E000C0006
001A000B000F0006000C0007
001B000B000F0003000200010000
001B000B000F0004000500060007
001B000B000F0000000100120013
001B000B000F0006000500140015
001B000B000F0013001200160017
001B000B000F0015001400180019
001B000B000F0016001900180017
001A000B000F0000001A0003
001A000B000F001A00000013
001A000B000F001A00130017
001A000B000F0018001A0017
001A000B000F001A00180014
001A000B000F0005001A0014
001A000B000F001A00050004
001B000C00B0001B00030004001C
001B000C00B0001B000900020003
001B000C00B0001C00040007000D
001B000C00B0001B00030004001C
001B000C00B0001B000900020003
001B000C00B0001C00040007000D
001A000C00B0000800010002
001B000C00B00008001100120001
001B000C00B00012001100100016
001B000C00B0000F001900160010
001B000C00B0000F000E00150019
001B000C00B00015000E000C0006
001A000C00B00006000C0007
001B000C00B00003000200010000
001B000C00B00004000500060007
001B000C00B00000000100120013
001B000C00B00006000500140015
001B000C00B00013001200160017
001B000C00B00015001400180019
001B000C00B00016001900180017
001A000C00B00000001A0003
001A000C00B0001A00000013
001A000C00B0001A00130017
001A000C00B00018001A0017
001A000C00B0001A00180014
001A000C00B00005001A0014
001A000C00B0001A00050004
0000
FFFF
002000200020

Shadow.3 (tfx3):

FF8B3E80000000007FFF00000000000000000000
0002000300030000000000000000000E039C039C039C0000000000000006
002701900386
002701880138
002701800128
002701780118
002701700108
0027016800F8
0027016000E8
0027015800D8
0027015000C8
0027014800B8
0027014000A8
002701380098
002701300088
002701280078
002701200068
002701180058
002701100048
002701080038
002701000028
002700F80018
002700F00008
000700C6
0000
0098FFFF
000700BC
0000
0098FFFE
000700B2
0000
0098FFFD
000700A8
0000
0098FFFC
0007009E
0000
0098FFFB
00070094
0000
0098FFFA
0007008A
0000
0098FFF9
00070080
0000
0098FFF8
00070076
0000
0098FFF7
0007006C
0000
0098FFF6
00070062
0000
0098FFF5
00070058
0000
0098FFF4
0007004E
0000
0098FFF3
00070044
0000
0098FFF2
0007003A
0000
0098FFF1
00070030
0000
0098FFF0
00070026
0000
0098FFEF
0007001C
0000
0098FFEE
00070012
0000
0098FFED
00070008
0000
0098FFEC
00620000000500000005
0066FFF6
0064FFF6
0066000A
006100040000
004200000224
00420001021E
004301FF0038
004300FF0212
0043007F020C
0043003F018E
0043001F014C
0043000F010A
0043000700C8
004300030086
004300010044
000701E4
0000
00620004FF8B0000FFFB
0066000A
006100020004
00470006000400000040003F0040003F007F0000007F
008800040004000500030002
000701A8
0000
00620004FFC70000FFFB
0066000A
006100020004
00470006000400000040003F0040003F007F0000007F
008800040004000500030002
0007016C
0000
00620004FFDD0000FFFB
0066000A
006100020004
00470006000400000040003F0040003F007F0000007F
008800040004000500030002
00070130
0000
00620004FFEA0000FFFB
0066000A
006100020004
00470006000400000040003F0040003F007F0000007F
008800040004000500030002
000700F4
0000
00620004001600000005
0066FFF6
006100020004
00470006000400000040003F0040003F007F0000007F
008800040004000500010000
000700B8
0000
00620004002300000005
0066FFF6
006100020004
00470006000400000040003F0040003F007F0000007F
008800040004000500010000
0007007C
0000
00620004003900000005
0066FFF6
006100020004
00470006000400000040003F0040003F007F0000007F
008800040004000500010000
00070040
0000
00620004007500000005
0066FFF6
006100020004
00470006000400000040003F0040003F007F0000007F
008800040004000500010000
00070004
0000
0000
FFFF
002000200020

* Note how tfx3 0042 opcode has a 0000 time value, used in company with 0043.

Tfx3 also added the 00A1 / 00A2 shadow format upgrade, however, these were only used, exclusively, for the highest LOD F22 models.

The 0043 opcode control over shadow rendering is fairly straight forward, but they don’t use it in that way. For example, if you check the very last vertex/ uv-mapping block in each of the building shadow blocks, you will see, at least in all the models I checked, it is never rendered. I suspect It is the block for the 00:00 shadow (value = 00) which reuses the 18:00 shadow (3F), ie. RED1800(3F) is reused for RED2000 (7F), MOONLIT NIGHT (FF) and NO MOON AT NIGHT (00). I believe the unused block, at the end of the texture mapping section, is written specifically for the 00:00 / NO MOON AT NIGHT (00) shadow.

(still working on the bit mask calculations)

Note that time values 00, 7F and FF are not included in the basic, static object files 0043 tree:

Build.3:

004301ff002e    ;

004301ff0028

0043003f0172

0043001f0134

0043000f00f6

0043000700b8

00430003007a

00430001003c

0000

 

I believe they tried to save some cycles, in shadow rendering, by modifications reusing the RED1800 shadow, with the 0043 shadowing process. They appear to be simply reusing the RED1800 shadow for 20:00, 22:00 and 00:00.

Point of information:  Previous difficulty with finding an in-game “NO MOON AT NIGHT” scenario, on which to test, is resolved by using the tfx3 diagnostics; it even prints a log, in the CCCW MFD, of the T.O.D. being used:

capture_001_31072017_135817.jpg.298862439430f62e8848fcdb10cce556.jpg

Share this post


Link to post
Share on other sites
On 5.5.2017 at 9:57 PM, Krycztij said:

Two points don’t define a plane … how does 0011 decide to jump, then? Maybe distance related (point 1 closer to the viewer than point 2)?

Wow nice to see you guys are still at work ^^

While studying for exams I've found a moment to check in on the forums here (procrastinating like a boss), and this just caught my eye, as I've been studying some linear algebra lately so here goes a short comment:

having an initial starting point A, and a second different point B, the vector from point A to B could be interpreted as a normal vector to a plane, couldn't it? so two points COULD actually define a plane..

sorry if this comment makes no sense in the context of the 0011 op code, I've only briefly been able to read through the posts, most of the stuff I just don't have a lot of time to fully comprehend yet... but yeah, just wanted to throw that out there.. ^^ keep it up guys!

Share this post


Link to post
Share on other sites

Oh man, linear algebra.  That brings back memories...

Actually, not really.  I didn't retain a whole lot from that class. :D

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now