You noted that you were worshiping a God who didn't do anything for you...but you had no expectations--or wish--that He would. Am I understanding that correctly?
Christianity begins with a love of God which begins the process of loving God's ways (laws) and serving Him. Judaism begins with loving God's laws and serving Him which leads to loving God.
Serving our fellow man is the way we serve God. Where mankind has troubles is when we wish to serve only ourselves and have no respect for Law.
An agnostic who seeks good (keeping the Law) and extends a helping hand to others, in my opinion, is on the path to loving God.