FE10: Radiant Dawn Hacking Notes

[HeartlessSeph]

On 7/13/2021 at 1:13 AM, nick0807 said:

I keep getting an application error when trying to run the data builder. I get an error saying "The application was unable to start correctly (0xc000007b)

That's weird, you're just drag and dropping the uncompressed file onto the exe right? It should have no issues starting afterward.

[dioxide_carbonite]

On 7/13/2021 at 7:48 PM, dioxide_carbonite said:

I'm feeling really stupid right now, but how do I know the internal ID of the time? I've tried searching for all the columns inside the shop file, and none of them return anything.
I saw some other posts about changing items, but I couldn't figure anything out. I know the pointers point to addresses (like where the item name is), but that didn't help me much.
I'm looking to edit the contents of the forge, and I'm lost. I know I'm in the right file (shopitem_h.bin), and I've successfully edited item stats, stat caps, and character growths before; but somehow this is eluding me. I am aware the forge contents are listed on a chapterly basis, so I might just edit the 4-5 forge and point everything at that.

I tried to edit my original post, but I guess there's a timer on that. ANYWAY. Something that was really confusing me is that the item ID changes in every file, so you can't reference other files to find the ID you need for this file. So for example in the FE10Data.cms file, the address for the Iron Sword is 0x2AC66 while the address for the Iron Sword in shopitem_h.bin is 0xAC31 then you'd need to subtract 20 from both for the ID. In the attached image, you can see when I search for 0xAC11 in the same column is 0xABD5 and 0xAB92 which are the Iron Lance and Iron Axe respectively. I wasn't really expecting them to be offset like that (and was expecting them to be back to back), but that was why I was having a bit of trouble finding them once I got the other IDs. It goes <item ID> <0x> <item ID> which just happened to line them up in columns, but there are others in the same row. Hopefully this post helps somebody else.

Edited July 19, 2021 by dioxide_carbonite

[HeartlessSeph]

9 hours ago, dioxide_carbonite said:

I tried to edit my original post, but I guess there's a timer on that. ANYWAY. Something that was really confusing me is that the item ID changes in every file, so you can't reference other files to find the ID you need for this file. So for example in the FE10Data.cms file, the address for the Iron Sword is 0x2AC66 while the address for the Iron Sword in shopitem_h.bin is 0xAC31 then you'd need to subtract 20 from both for the ID. In the attached image, you can see when I search for 0xAC11 in the same column is 0xABD5 and 0xAB92 which are the Iron Lance and Iron Axe respectively. I wasn't really expecting them to be offset like that (and was expecting them to be back to back), but that was why I was having a bit of trouble finding them once I got the other IDs. It goes <item ID> <0x> <item ID> which just happened to line them up in columns, but there are others in the same row. Hopefully this post helps somebody else.

Shop_item extractor coming some time by the end of the month or the beginning of next month. Soon there'll be no need to mess with pointers.

[thane98]

Hey all - I started working on Radiant Dawn support in Paragon. The formats are very similar to 3DSFE, so the only big changes were big endian support + decoding for TPL textures. With all of that implemented, you have full access to the module system just like FE13-FE15. Example:

I still need to write modules for a few files before I'll consider this ready to go, but that's the easy part. Hoping to have an initial release with editors like this for core data + a proper chapter editor. I may try a dialogue simulator as well, but I have a feeling that changed quite a bit going to 3DSFE.

 

Edited July 24, 2021 by thane98

[HeartlessSeph]

18 hours ago, thane98 said:

Hey all - I started working on Radiant Dawn support in Paragon. The formats are very similar to 3DSFE, so the only big changes were big endian support + decoding for TPL textures. With all of that implemented, you have full access to the module system just like FE13-FE15. Example:

I still need to write modules for a few files before I'll consider this ready to go, but that's the easy part. Hoping to have an initial release with editors like this for core data + a proper chapter editor. I may try a dialogue simulator as well, but I have a feeling that changed quite a bit going to 3DSFE.

 

Yoooooooooooo

You are based for this. Was going to ask to use your program as a base for my update but now I don't have to. 

[Vicious Sal]

19 hours ago, thane98 said:

Hey all - I started working on Radiant Dawn support in Paragon. The formats are very similar to 3DSFE, so the only big changes were big endian support + decoding for TPL textures. With all of that implemented, you have full access to the module system just like FE13-FE15. Example:

I still need to write modules for a few files before I'll consider this ready to go, but that's the easy part. Hoping to have an initial release with editors like this for core data + a proper chapter editor. I may try a dialogue simulator as well, but I have a feeling that changed quite a bit going to 3DSFE.

 

That is pretty damn amazing! 
Looking forward to testing this out. ^^
If you've any questions about RD and the code, there's a few people around here that have been working with RD for a while, and I'm sure we're all very happy to help!

[thane98]

On 7/24/2021 at 4:36 PM, HeartlessSeph said:

Yoooooooooooo

You are based for this. Was going to ask to use your program as a base for my update but now I don't have to. 

On 7/24/2021 at 5:14 PM, Vicious Sal said:

That is pretty damn amazing! 
Looking forward to testing this out. ^^
If you've any questions about RD and the code, there's a few people around here that have been working with RD for a while, and I'm sure we're all very happy to help!

Thanks!

I've been a little busy lately, but I finally got some time to work on this again. So far I've written modules for FE10Data, facedata, shopdata, and dispos. Paragon's also producing files that are almost identical to the originals now - only difference is sort order for text data with Japanese characters.

Finished a few more UIs as well.

Spoiler

Spoiler

Spoiler

This is almost ready to go - I want to add a few more modules first. This thread's been very useful for writing them so far!

Also, if anyone wants to help with writing modules everything is in the GitHub repo under the fe10 branch: https://github.com/thane98/paragon/tree/fe10

I could use a little help with chapter blocks if anyone has those mapped out. Paragon can parse them, but most of the fields have generic names since I haven't figured out what they do yet.

[1234567890num]

This looks amazing and might finally let us not deal with hex and pointers ^_^.

I made a PR that clarifies some of the unknowns with what I discovered but hidden within the thread including FE10Data.cms' ChapterData. I hope mess files and script files can have modules written soon.

[thane98]

4 hours ago, 1234567890num said:

This looks amazing and might finally let us not deal with hex and pointers ^_^.

I made a PR that clarifies some of the unknowns with what I discovered but hidden within the thread including FE10Data.cms' ChapterData. I hope mess files and script files can have modules written soon.

Thanks for doing that!

I'd like to support mess/dialogue on the same level as 3DSFE, but there are a couple things to work through there. While the syntax is the same, the commands are all different. I've seen some documentation on them in this thread, but for a release I'd like to have all of them mapped out. After that's done, I'll need to set up rendering in the simulator for FE10 dialogue. Main issue there is that my tpl parsing code is a little buggy - it works fine for portraits and icons, but fails to load several UI elements including talk windows.

For scripting, I have plans to support that in Paragon but it's not even there for 3DSFE yet. There's a standalone editor, Exalt, that can compile + decompile scripts, but that's on a different tech stack from Paragon. I've got a project to reimplement it in rust. Maybe FE9/FE10 support can go there.

[HeartlessSeph]

9 hours ago, thane98 said:

Thanks for doing that!

I'd like to support mess/dialogue on the same level as 3DSFE, but there are a couple things to work through there. While the syntax is the same, the commands are all different. I've seen some documentation on them in this thread, but for a release I'd like to have all of them mapped out. After that's done, I'll need to set up rendering in the simulator for FE10 dialogue. Main issue there is that my tpl parsing code is a little buggy - it works fine for portraits and icons, but fails to load several UI elements including talk windows.

For scripting, I have plans to support that in Paragon but it's not even there for 3DSFE yet. There's a standalone editor, Exalt, that can compile + decompile scripts, but that's on a different tech stack from Paragon. I've got a project to reimplement it in rust. Maybe FE9/FE10 support can go there.

There's a thread on FEUniverse that details the scripting format for FE10. It includes a quick python script that parses the cmb too. For now it only spits out the script elements in cmd but exporting to a json or other type of text file shouldn't be hard. The repacking would still need to be worked on though. Don't know how different the op codes are from 3DSFE but it could be useful for when you implement the scripting.

I have a script that extracts the chapter data but iirc I didn't have much labeled for it either. I'll check the repo and see what is labeled. I'll do a PR if I know any missing labels there.

[thane98]

On 8/9/2021 at 6:29 PM, HeartlessSeph said:

There's a thread on FEUniverse that details the scripting format for FE10. It includes a quick python script that parses the cmb too. For now it only spits out the script elements in cmd but exporting to a json or other type of text file shouldn't be hard. The repacking would still need to be worked on though. Don't know how different the op codes are from 3DSFE but it could be useful for when you implement the scripting.

I have a script that extracts the chapter data but iirc I didn't have much labeled for it either. I'll check the repo and see what is labeled. I'll do a PR if I know any missing labels there.

Thanks for this - those notes were a big help. The format is very similar with a few minor differences:

• 3DS order is header, functions, text data. GCN and DS format is header, text data, functions.
• GCN/DS use different function headers.
• 3DS has floating point numbers. Opcodes 0x1 to 0x1E are identical in both versions, but the numbering gets off since the floating point op comes right after the integer op.

Overall, not too bad. Added support in the Rust library here so there's now a CLI option to convert scripts to/from YAML for editing if you're willing to build it. I'll look into adding an editor in Paragon as well.

[dioxide_carbonite]

Well, I was hoping that with my knowledge of the shop file and forge file I would be able to add or replace things offered in the forge (why do we never get offered El tomes?!), but that doesn't appear to be the case. From what I've gathered, the structure seems to be: MDV_<Type> IID_<Name> MIK_<Weapon Type> (example: MDV_IR for Iron, IID_Ironsword, MIK_SW for Sword). To confirm this, I tried adding the wind edge into the forge. I didn't know this at first, so I just put it into the first empty spot. Turns out that was the spot for the slim sword, and I could see "slim" in the forge then got a crash. I saw there's a "thrown" section (which is for the javelin and handaxe), so I tried putting it there thinking I was clever. No dice. I also tried changing Fire to Elfire, and that also resulted in a crash. "Iron Blade is iron" I thought to myself before getting another crash...

So I don't know if the MDV is the problem here (and there aren't any extra ones), or if there's something else looking for it to be the specific item it is. Searching the whole shop file just for say "iron sword" sounds like torture, so I'd rather avoid that. I think I've gone through the thread and did find some helpful info on page 1, but that was for switching the forge with another chapter's forge which doesn't help here. If anyone knows anything about changing the contents of the forge, please let me know.

In the screenshot, "0xAF76" is the IID for the Wind Edge.

[HeartlessSeph]

2 hours ago, dioxide_carbonite said:

Well, I was hoping that with my knowledge of the shop file and forge file I would be able to add or replace things offered in the forge (why do we never get offered El tomes?!), but that doesn't appear to be the case. From what I've gathered, the structure seems to be: MDV_<Type> IID_<Name> MIK_<Weapon Type> (example: MDV_IR for Iron, IID_Ironsword, MIK_SW for Sword). To confirm this, I tried adding the wind edge into the forge. I didn't know this at first, so I just put it into the first empty spot. Turns out that was the spot for the slim sword, and I could see "slim" in the forge then got a crash. I saw there's a "thrown" section (which is for the javelin and handaxe), so I tried putting it there thinking I was clever. No dice. I also tried changing Fire to Elfire, and that also resulted in a crash. "Iron Blade is iron" I thought to myself before getting another crash...

So I don't know if the MDV is the problem here (and there aren't any extra ones), or if there's something else looking for it to be the specific item it is. Searching the whole shop file just for say "iron sword" sounds like torture, so I'd rather avoid that. I think I've gone through the thread and did find some helpful info on page 1, but that was for switching the forge with another chapter's forge which doesn't help here. If anyone knows anything about changing the contents of the forge, please let me know.

In the screenshot, "0xAF76" is the IID for the Wind Edge.

I didn't test it, but Thane's FE10 paragon branch has forge data and shop data. Might want to check it out.

 

9 hours ago, thane98 said:

Thanks for this - those notes were a big help. The format is very similar with a few minor differences:

• 3DS order is header, functions, text data. GCN and DS format is header, text data, functions.
• GCN/DS use different function headers.
• 3DS has floating point numbers. Opcodes 0x1 to 0x1E are identical in both versions, but the numbering gets off since the floating point op comes right after the integer op.

Overall, not too bad. Added support in the Rust library here so there's now a CLI option to convert scripts to/from YAML for editing if you're willing to build it. I'll look into adding an editor in Paragon as well.

No problem, glad to contribute. I compiled the Paragon and it's a godsend. Going to start writing modules for FE10 Battle but there's a ton of unknowns in there. Can't thank you enough for your work.

[dioxide_carbonite]

On 8/7/2021 at 1:43 PM, thane98 said:

Thanks!

I've been a little busy lately, but I finally got some time to work on this again. So far I've written modules for FE10Data, facedata, shopdata, and dispos. Paragon's also producing files that are almost identical to the originals now - only difference is sort order for text data with Japanese characters.

Finished a few more UIs as well.

  Reveal hidden contents

  Reveal hidden contents

  Reveal hidden contents

This is almost ready to go - I want to add a few more modules first. This thread's been very useful for writing them so far!

Also, if anyone wants to help with writing modules everything is in the GitHub repo under the fe10 branch: https://github.com/thane98/paragon/tree/fe10

I could use a little help with chapter blocks if anyone has those mapped out. Paragon can parse them, but most of the fields have generic names since I haven't figured out what they do yet.

Sorry if this is obvious (I've never successfully built an application though tried a few times), but I'm stuck on this error:
I believe I did the previous steps correctly to get this part to even fire (required a restart).

[thane98]

1 hour ago, dioxide_carbonite said:

Sorry if this is obvious (I've never successfully built an application though tried a few times), but I'm stuck on this error:
I believe I did the previous steps correctly to get this part to even fire (required a restart).

You've done nothing wrong here, this error is my fault. I use a local version of mila for development, but the Cargo.toml in the repo is always supposed to point at the version on GitHub. I mistakenly committed my local one.

I've pushed in a fix, so it should work if you pull and build again: https://github.com/thane98/paragon/commit/afc94a1b33b7bf44ee6d9420799907eec34853a9#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542R20

[HeartlessSeph]

3 minutes ago, thane98 said:

You've done nothing wrong here, this error is my fault. I use a local version of mila for development, but the Cargo.toml in the repo is always supposed to point at the version on GitHub. I mistakenly committed my local one.

I've pushed in a fix, so it should work if you pull and build again: https://github.com/thane98/paragon/commit/afc94a1b33b7bf44ee6d9420799907eec34853a9#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542R20

I caught this error too, I just downloaded the versions of Mila and exalt and put them in My Documents/Github folder to resolve the error.

By the way, there's a key error for the FE10 Class Module. I edited the modules to fix it but didn't do a pull request since I figured those errors would be fixed by the time an official release build is public. 

[thane98]

11 minutes ago, HeartlessSeph said:

I caught this error too, I just downloaded the versions of Mila and exalt and put them in My Documents/Github folder to resolve the error.

By the way, there's a key error for the FE10 Class Module. I edited the modules to fix it but didn't do a pull request since I figured those errors would be fixed by the time an official release build is public. 

What’s the error? If it’s too small for a pull request I can just make the update.

Edit: Oh, I see. I merged a PR with some changes to the type but the UI is still referencing the old fields. I’ll fix this in the morning.

Edited August 15, 2021 by thane98

[dioxide_carbonite]

22 minutes ago, thane98 said:

You've done nothing wrong here, this error is my fault. I use a local version of mila for development, but the Cargo.toml in the repo is always supposed to point at the version on GitHub. I mistakenly committed my local one.

I've pushed in a fix, so it should work if you pull and build again: https://github.com/thane98/paragon/commit/afc94a1b33b7bf44ee6d9420799907eec34853a9#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542R20

$ python main.py
Traceback (most recent call last):
  File "main.py", line 5, in <module>
    from paragon.model.configuration import Configuration
ModuleNotFoundError: No module named 'paragon'

I tried the super simple "import paragon", but that just moved the error up to that line.

 

[thane98]

2 minutes ago, dioxide_carbonite said:

$ python main.py
Traceback (most recent call last):
  File "main.py", line 5, in <module>
    from paragon.model.configuration import Configuration
ModuleNotFoundError: No module named 'paragon'

I tried the super simple "import paragon", but that just moved the error up to that line.

 

Execute it from the root of the repo. The full command should be something like “python3 paragon/ui/main.py” (depending on what “python” points to for you since it looks like you’re on a Unix system).

[dioxide_carbonite]

2 minutes ago, thane98 said:

Execute it from the root of the repo. The full command should be something like “python3 paragon/ui/main.py” (depending on what “python” points to for you since it looks like you’re on a Unix system).

I tried both "python" and "python3" and got the same error from paragon-main.

[HeartlessSeph]

8 minutes ago, dioxide_carbonite said:

I tried both "python" and "python3" and got the same error from paragon-main.

Did you install Rust and all the dependencies listed in requirements.txt?

[dioxide_carbonite]

19 minutes ago, HeartlessSeph said:

Did you install Rust and all the dependencies listed in requirements.txt?

I did, but I guess I needed to build it again? It seems to be working now after rebuilding.
Edit: but it is missing the FE10 stuff
I tried copying over it from the earlier one, but it wouldn't show up.
Edit again: Went and got a fresh install, and now I'm getting 2 errors. Yes I rebuilt again and double checked the requirements.txt file.

$ python paragon/ui/main.py
Traceback (most recent call last):
  File "paragon/ui/main.py", line 5, in <module>
    from paragon.model.configuration import Configuration
  File "/home/[REDACTED]/Downloads/paragon-main/venv/lib/python3.8/site-packages/paragon/model/configuration.py", line 6, in <module>
    import pydantic
ModuleNotFoundError: No module named 'pydantic'

 

Edited August 15, 2021 by dioxide_carbonite

[HeartlessSeph]

48 minutes ago, dioxide_carbonite said:

I did, but I guess I needed to build it again? It seems to be working now after rebuilding.
Edit: but it is missing the FE10 stuff
I tried copying over it from the earlier one, but it wouldn't show up.
Edit again: Went and got a fresh install, and now I'm getting 2 errors. Yes I rebuilt again and double checked the requirements.txt file.

$ python paragon/ui/main.py
Traceback (most recent call last):
  File "paragon/ui/main.py", line 5, in <module>
    from paragon.model.configuration import Configuration
  File "/home/[REDACTED]/Downloads/paragon-main/venv/lib/python3.8/site-packages/paragon/model/configuration.py", line 6, in <module>
    import pydantic
ModuleNotFoundError: No module named 'pydantic'

 

You need to install pydantic. Try pip -install pydantic in Powershell or Cmd (Administrative Access for either one).

Edit: Guess I need to read a bit slower. Saw you installed everything from requirements.txt. Try installing after going in the virtual environment and seeing if it works then.

Edited August 15, 2021 by HeartlessSeph
Clarification after correctly reading post.

[thane98]

I tested the build on my Linux setup just to make sure I wasn't going crazy. It's working for me:

Spoiler

If it was working but not showing the FE10 option, you probably built the "main" branch. FE10 support is currently on the "fe10" branch.

And yeah, for dependency issues most I can suggest is deleting the virtual environment and trying again. I'm guessing the issue is something like installing on python 3 and then running on python 2, but difficult to tell here.

[dioxide_carbonite]

I got everything working on the first try somehow. Thanks for the help!
Now let's hope it can do what I wanted.
I'm guessing it's not supposed to give an error trying to open items, or has that not been added yet?

Edited August 16, 2021 by dioxide_carbonite