What are some starting weapons that are too useful for their own good...

The starting sword from all the dmc games.

The cane is so overrated. As a trick weapon, it's a really cool concept, but it just doesn't deliver any punch compared to the rest. Apparently it's effective for larger bosses since it can hit multiple weak points, but I've never had luck with it.
Saw Spear is my favorite early weapon. low stamina, high damage, and great reach in it's spear form.

Amygdala's arm was quite bad desu

This

Well yes, but the saw cleaver might actually be TOO good, its insanely quick attack can make a joke of some bosses that would otherwise be a struggle

also serrated on both sides unlike the cleaver

Saw Blade+Beast-Blood Pellet+Fire/Bolt Paper absolutely shits all over the Axe in terms of DPS, due in large part because the Saw's transformation attacks build up the BBP meter extremely quickly

I've been replaying Hyper Light Drifter, and the starting pistol just shits on all the other guns.

Also incredibly low stamina drain for the amount of damage it can output, forgot that

Definitely the most powerful weapon when maxed out. It helps that you get a shit ton of ammo for it too

You would only say all DMC games if you're forgetting about Alastor in DMC1. No human being ever keeps using rebellion after getting Alastor.

I like that it was an achievement to go through the whole game with just the plasma cutter, as if that was a challenge

Zweihander in Dark Souls 1. 2-hand R1 to victory.