Niceness and non assertiveness aren't the same thing. I think that is an important thing to bear in mind because if you are assertive, then that doesn't mean you are being not nice. If you think being assertive is the same as being unpleasant, you won't even try to be assertive. So you might be non assertive out of not wanting to upset people, but definitely don't frame it as being nice.
Imo, by being non assertive you just aren't being nice to yourself, and that person is the one you need to be the nicest too.
Yes, it sounds like you should work on being more assertive. But its going to take a lot of work and effort, and will be uncomfortable as ****. Being assertive is much easier if you are that way from the outset with people, as they learn behaviours based on how you interact with them, and you have to try to undo that now. I have done it myself, but with some people it takes a lot of work to "retrain" them. Some people it's ok, other people I literally just gave up on the friendship because what used to be acceptable to me wasn't any more, and they refused to adjust their behaviour.
Decide on boundaries early with people, assert them early on, and you won't have to later.