Progress here, of sorts.
I've been working with the Cap'n Binky tonight and I've managed to repair the Nexus 5k animation file bug by decompiling / recompiling one of the DLLs with a few changes. As such I'm not sure I'd be able to distribute the DLL itself with the fix, as it would probably be against the EULA of the tools.
but even so, the fix in itself is not the end of the story.
It has shown us that the Nexus art tool is spitting out a gr2 containing the model and separate gr2 files containing each of the anims. The thing is though... these load and display in Granny Viewer perfectly. And the produced anims don't play within the Nexus art tool. Seems that only the root bone is animating within Nexus even though they play perfectly in granny viewer.
Since the game assets
don't work in Granny Viewer, this leads us to believe there is some additional process required to turn the exported data from the Nexus granny FBX -> gr2 process to make it not only game ready, but working in the Nexus asset viewer properly.
As an experiment I did some more fiddling and managed to run Catherine's model gr2 and one of her animation gr2 files back through the process, as if she'd come from an FBX, and low and behold:
Looking at the binary contents of the two gr2 files for her model side by side they are massively different. So I think there's definitely another process that needs doing to these files that a) makes them work in the game and asset viewer and b) subsequently means they are no longer valid granny viewer files.
Will keep you all posted if we manage to figure any more out.
EDIT: Turns out Catherine's reprocessed model AND anims work fine in Nexus, so we just need to figure out why custom animations on custom models don't seem to work properly.