You can get away with very little supplies, actually. Basically just a small shovel, the rest be salvaged.
Pots can be made from old plastic bins/containers, soil/fertilizer can be made from food waste using worms, seeds can be made from surprisingly many fruits/vegetables. Pumpkin seeds are right inside the pumpkin, potatoes and beans can be put right into the soil, even tomatoes can be grown from store bought ones.
And while it's a really cool hobby: you're right regarding the cost effectiveness. Unless you happen to have a significant plot of land, it won't make a dent in your grocery bill.