Ok so first let’s go over what YouTube provides:
Storage, community tools, search algorithm, add sense, authority over copyright, front end.
Realistically you could probably cover the front end, search algorithm, and community tools with FOSS collaboration.
Everything else gets harder.
For storage, the VAST swaths of data, and forever growing nature of YouTube storage nearly guarantee its market dominance alone… if they can contain that infinitely growing monster forever. Its their greatest strength and can also be its Achilles heel. I would propose that video hosting would be covered by the creatives. This change creates a ripple effect that effect all the other challenges, but immediately raises the bar for entry, and with the exception of the highest earning creators, videos would have to be cycled out when their earning capability falls below cost to host. But! This has good sides, like the best videos would linger and bad videos would fall off increasing the quality of what remains. Creatives would have more control over their videos. You could also have a system that rotates videos between a cold storage and live videos, where cold storage would use a torrent like system vs the streaming of a live system, which would allow cheap storage of low earning videos to still have them available for those who could wait.
Copyright, so with the creatives holding the keys to the content, this new youtube would only facilitate the connection and front end, but would not regulate it. So copyright claims would have to be handled by the creatives. This is a sharp as hell double edged sword! You won’t be copyright trolled as successfully any more BUT your odds of ending up in court could be higher as there is no way to appease the record labels and what have you so readily.
There would also not be a method to scan the videos to easily find other people who are stealing YOUR content either. And you would have to deal with the person stealing your content directly.
And ad sense. Without a unifying front to bargain with advertisers, it will be like the Wild West. Most advertisers don’t have assurances of enforced standards and will be very timid to employ this new system. They would all have to vett creatives separately, and it would work allot like Sponcers do now, but ultimately i think it would be a boon, but for a wile the money won’t be there.