Why is it in popular imagining, that ghosts are {MOSTLY} depicted as miserable lonely sad folks that are stuck to the places they've died in?

I believe it's tied to the ides of the restless dead.

The spirit should move on after death, but some spirits get stuck between worlds. Often it's related to the circumstances of their death. Maybe they had unfinished business, they died a particularly grousom death, or they were denied a proper burrial.

If you have ever had a dream where you can't get something right. Like, in your dream you know you need to do something - you even know exactly what it is that you need to do. You have no excuse for not doing it - maybe you actually try to do it - and yet you keep finding it is as yet undone.

You eventually wake up, because you are alive and you are dreaming. But if you are dead, you are not alive, you cannot wake up. You have all of the agency that you had when you were dreaming, which is none. Until you woke up - you were helpless in your dream, try as you might. The dead are helpless in their dream as well, but they do not have the luxury of waking up.

So have some pity for the dead.

I don't believe in ghosts or psychic phenomena, but I do love the concept in fiction that ghosts aren't actual human souls. Rather, they're a sort of psychic "burn in." If a living person experiences strong emotions, such as a prolonged period of grieving, or the incredible emotional intensities that come with being murder victim, those emotions can become embedded within a place. Do you grieve for a deceased partner, mourning for years, remembering key moments over and over? A reflection of that grief becomes embedded within reality in the location you experienced those emotions. When you die or leave, someone else can come into that place and experience a recording, a reflection, or echo of the emotions and memories you experienced.

Ghosts are effectively traumatic memories burned in to the fabric of the world. They don't actually experience anything; they're not conscious beings. They're not souls looking to complete their business and move on. They're simply psychic echoes. They're imprints left on reality from very intense and painful emotions, particularly those experienced repeatedly over a long period of time.

This also explains why ghosts have a half life. Ever wonder why in the US, all the ghosts seem to be old timey white people from the 1800s or similar? Considering the total number of Native Americans that must have lived in what is now the US down the millennia, the vast, vast majority of ghosts should be Native American. But aside from the classic example of a disturbed native burial ground, Native American ghosts don't show up much in fiction. It's usually old timey white people.

The reason for this, in the imprint theory, is that like any imprint, ghosts tend to fade with time. Just as most footprints will slowly be eroded, the knots in the psychic fabric that ghosts represent slowly work themselves out over time. The ghosts people do experience tend to be from the last century or two, as most ghosts older than that have decayed below the level of human perception.

I think because in that framing being a ghost is a sad lonely thing. In the Christian tradition the ultimate reward is heaven and being in the presence of the lord. If one is stuck on earth it is similar to catholicism's concept of purgatory.

I'd imagine that the ghosts who ditched their unfinished business to explore the world beyond would be doing that exploring in a world beyond so we wouldn't see them or anything.

The traditional conception is that ghosts aren't really sentient creatures, but a phenomenon that souls can get stuck in instead of going to the afterlife. There's symbolism there - a ghost is an echo of a person, much like memories of them or the effects they had on the world. There's also a bit of Christian moralizing - a "good" person doesn't stick around because they are eager to join God in heaven, while a "bad" person clings to their earthly life and possessions even if they are only capably of doing so in a greatly diminished state.

The modern conception of a ghost where it's a fully realized person who can just kinda go through walls is an anthropomorphized and secular version of the ghosts that were invented by the Victorians.

Being a ghost is not supposed to be the final stage. Some have unfinished business right where they are but they’ve been waiting for ages and that makes them sad.

In anglo-Christian culture ghosts were explained away as souls in purgatory, and this led to a Catholic/Protestant split wrt whether they existed or not, because Prots denied the existence of purgatory. They are still religiously problematic really — think about how often the 'ghosts' in American media turn out to be demons in order to keep things in line with scripture.

I personally believe there's something much more terrifying implied in the English folk-tradition though: a spirit bound to the last physical vestiges of their time on earth, going through the motions as what little is left of their mind after the trauma of death unravels completely. They're dead — it's over. They can still do the things they did in life, but it doesn't mean anything anymore.

Obviously there are very powerful resonances here for anyone who has witnessed a person sicken and die.

Alright, if they're having a romp in the afterlife in some capacity, they aren't ghosts. There's a kind of underlying implication they're sticking around for some worldly purpose or reason. Often this is used for metaphor about moving on in your life (or rather afterlife).

Because we think of every life as a story, and every story needs narrative closure. So if someone’s life seems like an unfinished story we feel like there’s some kind of lingering agency trying to finish it.

Because they are old. Ghosts are just the anthropomorphic manifestation of people's fear of growing old. Religious framings are just an add on.

Trauma and grief can't run their course if your mind is so senile and your short term memory so feeble, that you're basically forced to live in the past. Forever repeating old arguments, reliving past trauma and never overcoming old fears. With your mind so set in it's tracks, that you can't even imagine leaving the place where you lived all your live — your "old haunt" so to speak. How could you live in the present, if you can't even recognize your own children half of the time? But the long term memory often still works. Ghosts are real and if you're lucky enough to live that long you might well become one. Of course aging isn't always like this, it can be graceful and dignified but when it isn't, that's what people are afraid of.

People are scared, when they see older relatives acting stranger every day, especially in times before any way to diagnose Alzheimer's and other forms of neural degradation. They might seem like they are not quite here anymore, like the person they were had long since died and yet, something lingers. Ghost stories are a socially acceptable way to express those fears.

Just observe the effects ghosts have on their victims: first, they are reminded of their own mortality. Then their hair suddenly turns white or gray or falls out, they lose sleep, wake up tired or grow old over night. They might lose their mind or die themselves. That's all just normal aging.

Here is a handy key to select monsters and their meaning:

• ghosts 👻: aging, death, old people
• vampires(folk believe): plague, infection
• vampires(literature): landlords, feudalism
• zombies(modern): alienation, capitalism
• witches: women who stand up to patriarchy
• Frankenstein's monster: the proletariat gaining class consciousness (no seriously)

I recommend the podcast "the horror vanguard" for details.

I think in some asian portrayals of ghosts, its because they have "unfinished bussiness" so their spirit wont leave.

Popular beliefs influences people's beliefs, which reinforces popular beliefs. Step back even farther from the question for a moment and ask, "why do you think of ghosts as dead human spirits at all?" That a "ghost" is some sort of dead human spirit is a concept that has been built into Western society for a long time. It is something we just accept in story telling and mythological belief systems because it's been in them so long and is told to us via authoritative figures in our lives from an early age. To tell a story where a ghost is anything other than a dead human spirit or the echo of a dead human, makes people call bullshit on the story, because the story has broken a long standing societal expectation. Sure, some stories can get away with it, and more so in the modern age where we are starting to appreciate stories which subvert long standing expectations. But, we still tend to fall back on old tropes and devices which we can expect readers to understand, without having to spend too much time on building a world. It's far easier to save the term "ghost" for something much like a dead human spirit and just create a new term when trying to describe something else.

Sour grapes.

There is nothing popular fiction hates more than somebody doing something everyone wants to do but can’t. Impossibility, when possible, becomes cast as immorality or immaturity or otherwise something arbitrarily undesirable.

To be a ghost is depressing and/or monstrous because when we die in real life we don’t stick around. Time travel overwrites reality with a worse version of the present because in real life we can’t change the past. Resurrection brings people back as monsters because in real life we can’t have our lost loved ones back. Immortality is sad and lonely and often requires you to do evil things to sustain it because in real life we can’t live forever. Traveling to alternate lifetimes where you’re more successful is emotionally hollow because you had the most important emotional stuff in your life all along and you wouldn’t trade that for the world.

These and other speculative crises always have to be fixed by making the fictional world abide be the limitations of the real one. Aren’t we so lucky that our world is randomly already like this?

Most of the ghosts at Hogwarts in Harry Potter seem to either actively enjoy being a ghost or at least not mind it and just carry on doing stuff (like assist the pupils)

That aside, plenty stories have the ghost relive the moments before their death involuntarily, over and over. So for them it's nothing they can control.

But I'm with you: If I were not trapped in this deathloop (or bound to one location) and were able to be seen by other humans, I would definitely not be gloomy, once I got over the fact that I died and can no longer interact in all the usual ways with my spouse and dogs. Even if I could not do crazy telekinetic stuff, I could at least wander nature, haunt bad politicians and give my spouse inside knowledge, which she can possibly make a lot of money from to buy a house for us.

Disclaimer that I don't believe in ghosts, but in fiction at least, I think the usual implication is that a ghost is someone who hasn't passed on correctly. A few people have brought up unfinished business already, but even in stories that don't bring that up, ghosts are often people who died horribly, prematurely, and/or violently. Sometimes they're explicitly under a curse keeping them from moving on.

Basically, the circumstances of their existence are wrong, and they're stuck due to forces beyond their control. That's kind of the tragedy of being a ghost; they're often a whittled-down, corrupted version of their living self.