He'd still have the 30 km per second (67,000 mph) of sideways velocity that Earth has when it's orbiting the sun, and that speed is enough to prevent Earth from ever hitting the sun.
Orbits aren't very intuitive; if you want something you launch from Earth to fall straight into the sun, you actually need to fire it directly opposite to the direction that Earth orbits in. So if you imagine Earth orbiting clockwise, you want to shoot the thing counterclockwise. If you do that at the right speed, you counteract all the orbital speed and the thing just falls into the sun.
If you can speed something up a completely unlimited amount then sure you could aim straight at the sun and just fire it so fast that it hits the sun anyway. It'll be off-centre a bit but the sun is pretty big. Consider how much of the sky isn't sun though. If the sun is directly overhead and you shoot straight at the sun, the thing you fired is already going 30 km/s sideways before you even started. We could do the trig to figure out how fast you need to shoot it to still hit the sun anyway but I think the more important part here is getting a feel for the motion involved.
