Deltre

I'm surprised you found a version of the Nightmare pack without the Terrain Editor! I don't have a copy on hand but it's one of the newest modules so maybe just search for "new nightmare module FE8" or something like that.
The Staff EXP Constant is located at 2C676, however.

Okay, so admittedly I haven't messed around with this stuff in a few months, and I completely mis-remembered the process for editing the nightmare module manually. My bad man. So actually, you should be good just by changing the first number to the desired number of classes. DO NOT change the second number, it needs to be 84. Again, my bad, I realized as soon as I opened the nmm file for myself.
I do have to ask, are you expanding the ROM or the table? Those are two totally different things. One will actually increase the overall size of the ROM (you don't want this) where as the other just moves the data to an area with more room. If you do actually expand the ROM you'll probably run into compatibility issues with Nightmare/FEditor and honestly you'd have to add an incredible amount of content on top of the existing content in order to really justify ROM expansion. Even worse, you'll break literally everything that comes after the spot you paste to in the ROM, so make sure you always Ctrl+B, not Ctrl+V.
Now for actually getting the editor to work, I'm 99% sure that your problem is the line that says 0x807110. That looks like the default class table position, whereas it should be pointing to wherever you moved the class table to. That said there's good news and bad news. The good news is that it's really easy to move a table to free space. This is where everything I described in my second bullet point comes into play. Open up your ROM in a hex editor of your choice (I like HxD but it really doesn't matter that much) and search for offset 807110. Then take the default 128 (80 in hex) and multiply by 84 (54 in hex). This will give you a total of 2A00 (again, in hex). Starting from 807110, select 2A00 bytes of space (most editors should have a feature like this, HxD will do this with the 'select block' command) and copy. This is your class table. Ctrl+B (NOT Ctrl+V) this into free space somewhere. I moved mine to B35670, for example, although anywhere with free space (look for long strings of FF) will do. Now in the text file, change the line where it says 0x807110 into wherever you pasted the class table. In my case, it would say 0xB35670. From there you can add classes to your liking, no problem.
The bad news is that if you already added a new class chances are you overwrote something else. In the default position there's not really any room for another class, so the game would be reading a combination of your new parameters as well as whatever data was already there. That's most likely what's causing the glitches you're describing. If your ROM starts acting strange even after you move the class table, it may be time to start on a backup ROM.
The only other thing that could affect this is how you replaced the pointers. This is pretty simple. So we know that the default pointer is 0x807110 based on the nmm. You have to reverse the order of the bytes (not the numbers themselves) so that 80 71 10 becomes 10 71 80. Then, slap an 08 on the end, giving you 10 71 80 08 as the hex string you're replacing. Then take whatever offset you pasted the table to and do the same. In my case, I pasted to B35670, so my new hex-string should be 70 56 B3 08. I would run a find-replace command for 10 71 80 08 and replace all instances with 70 56 B3 08. Assuming I've done everything else correctly, it should work.
I don't really art so I have no idea about the battle palette question. I might be able to help you with the animation issue but I'm having a little trouble understanding what you're asking.

Hoo boy, that sounds like a lot of hassle haha. Although I believe that process should work in theory, EA is just so much easier to deal with in my opinion, and there's much less room for error once you understand what you're looking at. So here's a breakdown of what to looking for in EA instead.
The first thing I'm going to recommend is that you reassemble the entire chapter together instead of just the unit list. While that will work (usually) I've had some strange reactions before such as units not loading, cutscene corruption, etc. where as by reassembling the entire chapter I can be more confident that things will work as planned. This is particularly true if you want to start messing with other things besides the Unit List, but that's just my preference.
Okay so you've got the offset of the chapter you want to disassemble (in this case $9E863C), and you've disassembled it with EA into a text file. If you follow my advice to reassemble the whole chapter I recommend that you move the Pointers List to the top of the file like so
Then you'll want to scroll down a bit until you find the until you see the enemy unit list. Units1 is to the best of my knowledge reserved for player units, so it'll be something like labelXX. For the prologue, it looks like this
You can see that I've already added an archer to the list, but FE8 is a little weird sometimes when you just throw a unit into a pre-existing list, even when you've moved everything to a place with ample free space. Let's look at just one unit.
UNIT 0x68 0x3F 0x0 0x24 [14,8] 0b 0x0 0x1 0x8B3BFC [0x20,0xBD,0x0,0x0] [0x0,0x12,0x0,0x0]
UNIT tells the game that this is well...a unit. The next four bytes (in order) are Character ID, Class ID, Leader (I believe anyways, I never use it and this is usually 0x0), and Level. Following that, we have the co-ordinates on the map where this unit will appear (this is the initial position by the way). The next byte is basically a flag that tells the game if the unit in question has any kind of special behavior associated (such as drop last item, randomized placement for skirmishes/tower, etc.). This is followed by two bytes that I am unsure of off the top of my head. Following that is a pointer to a unit's REDA (REinforcement DAta) which will be very important, but I will come back to that. Not every unit will have REDA, but you should take note of it in Unit Labels that do. The remaining eight bytes are the unit's inventory (first four), followed by their AI behavior (last four). For the sake of my example let's say this is the only unit in the chapter. I relay this information to the game by putting another UNIT at the end of my unit list
label36:
UNIT 0x82 0x19 0x0 0xD [14,7] 0b 0x0 0x1 archer [0x2D,0x0,0x0,0x0] [0x0,0x12,0x0,0x0]
UNIT
like so.
As for the REDA, I'm guessing that this may be the reason that you're not having any success with your own method. Let's look at the archer I added.
UNIT 0x82 0x19 0x0 0xD [14,7] 0b 0x0 0x1 archer [0x2D,0x0,0x0,0x0] [0x0,0x12,0x0,0x0]
So if we take this Archer's co-ordinates at face value, we would assume that he appears on the map at [14,7]. Already there's a problem, as he shares this position with two other units! However, if you load the game up, you'll see that in reality he's standing at [12,9] in battle. First he appears at [14,7] as seen in the cutscene, then moves to the position defined by his REDA, in this case, [12,9]. Now, the reason I'm giving this part so much detail is that in my experience, adding more units to a list that has REDA means that you must also assign REDA for any new units. I haven't really found a way around this, as even under a separate label for new units the game still seems to requires REDA even if the initial position is going to be the same as the REDA position, but I digress. Fortunately the REDA itself is very simple.
There's not really much to explain. All we really have is the label name (archer), the command (REDA), and the coordinate [12,9]. Following that we have four bytes that I don't really remember; look at a dedicated tutorial if you have to know but it's nothing too crucial for what you're trying to do. Hopefully that will at least get you started man :D
Also, it's worth knowing that everything I said about REDA's only applies if there's at least one pre-existing REDA pointer on that particular unit list. If there's no REDA to begin with, you can just add units freely and their assigned coordinates will work properly. It's a mystery

Open the class editor and change the part that says "Battle Animations Pointer" to NEHU instead of NDHU. This will let you put in any hex value instead of choosing from a drop-down menu, although you'll have to keep a reference somewhere as to which pointer leads to which animation. I kind of suspect that isn't the main problem though; you said you essentially replaced the Twins' animations correct? I would definitely recommend adding the animations to any unused slot in the future, just to avoid potential issues. Also, it's worth noting that you shouldn't edit in both Nightmare and FEditor at the same time. Since they basically take a snapshot of the ROM, it's very easy to overwrite changes made in one with the other if you aren't taking special care, leading to a lot of situations where a change "doesn't work."