It is only if the primary or only reason you do what you do is to make money that you will envy every random person who made or makes a lot of money (or money that exceeds what you made or make).
You are self employed. You work to pay your bills, build your dreams and create your life, regardless of who signs your paycheck.
If self-employment is the opposite of employed-by-other, than a job is outsourcing your freedom (time and talent) to a boss.