Just to point out that you stated that factory farming is bad for the environment, but it is actually the most sustainable form of animal agriculture with respect to the environment. With factory farming the entire process has been optimized for cost (don't forget the entire point of the animal agriculture industry is to make money). What is not prioritized is the animal welfare, in any regard. In the stereotypically rich countries such as the UK, Switzerland, the US and Australia it is still standard practice to gas pigs to death, similar to the gas chambers used in WW2, send all male chicks to death through a giant macerator and other truely horrific forms of torture and death.
If you then look at the other side, in where there are animals in green fields living "happy lives" then this has the greatest, by far, impact on the environment. The amount of farming land needed to sustain the current world-wide meat and dairy consumption using the non factory-farmed animals far far exceeds the available land mass that we have, which is the reason factory farms exist. Where there is a demand, there needs to be a supply.
And when it comes to the difference between plants and animals, there is a fundamental difference that we need to consider - do plants suffer? As most (sane) people are against animal cruelty we need to ask ourselves what that actually means. Most people would answer that they don't want the animals to suffer. The reason we, and animals, suffer is because we have a consciousness and have the ability to feel pain through our nervous system - plants do not have this ability. So when you compare pulling a carrot from the ground to slicing a cows neck where the cow screams and thrashes and clearly does not want to die - it's pretty disingenuous.
My last point is that you make the statement that due to us having historically eaten meat and dairy that implies that gives us justification to continue. There are many horrible things humans have done in history (kill, rape) and we have built our current ethical and moral viewpoints from constantly improving, and striving to improve, as a species.
Most of us lucky enough to even be here in this small community that are likely the people that live in wealthy enough countries where there is a choice we can make 3 times a day to reduce causing needless suffering to these animals. Where I live in Europe, and many other countries now, I can simply buy plant milk instead of cow milk when eating my cereal, or one of the many plant alternatives. And of course this doesn't apply to everyone on the planet, some people are forced to consume animal products due to the system they live in but for most people reading this there is a choice.
I strongly recommend watching this documentary to see what the animal agriculture industry truly is:
Dominion (2018) - full documentary
It's quite late where I am but tomorrow I will edit and add sources to my claims.