Sounds like you've got one or two things wrong here.
Your starting party is defined in the System tab, not the Actors tab. If you don't change the system tab and you just delete actors from the Actors tab to get rid of anyone you don't want in your party at the start of the game, you still have them all in your party - they just don't have sprites. So you'll have your actor and three "blank" followers. When you add a new party member, they will be #5 and will not show up on the map. In the menu, you'll have your leader, then 3 blank spaces, then your new party member. If you don't want 4 actors in your party at the start of the game, you delete the unwanted ones from the System tab, not the Actors tab.
The other thing is that when you change actors who are in your party, or change the starting party on the System tab, you have to start a new game for those changes to be picked up. Your starting party is obviously only relevant at the start of a game, so if you make changes to that, then load a saved game, it's not going to have any effect. And actors who have joined your party have all their data saved when you save a game, so if you save, make changes to actors, then resume your saved game, your party members will be loaded from the save file, not from the database. The only time they're loaded from the database is when you add them to the party the first time, or when you add them with the Initialize box checked.