Have you ever watched a video where you couldn’t understand a word of what was being said? Maybe the volume was too low or the people were whispering. Perhaps you are hard-of-hearing or even deaf. Or it was just in a foreign language. These are no reasons to be ashamed! You’re only one click away from seeing what was being said using the CC button!
Today, the closed caption “CC” is one of the most recognizable icons in the world. A good part of that is due to the advent of captioning for online video and streaming services. In particular, YouTube once stood out as an early adopter and strong advocate for closed captioning.
Alas in recent years they have made decisions that have led to them going from caption-hero to villain. Their latest act of treachery is their decision to remove community contributions, a feature that allowed viewers to submit captions for videos they wish to transcribe or translate.
As a sort of countdown, until community contributions are gone for good, I am going to be recounting the history of closed captions on YouTube, over the next few weeks. I hope our readers will find this to be a fascinating retrospective on a much-overlooked technology…
Our story begins in 2005, with the inception of two websites: YouTube and Google Video. Now, online video was by no means a new phenomenon, the concept of a viral video was already a decade old at this point. But the video culture of the time was quite different. Videos were big files, so if you wanted to share a clip with your friend via email, you needed to lower the quality and duration. If you wanted to have it available for the world to see you would need to host it on a web server and even then cost would be the least of your worries. What had been a daunting challenge up until this post was building a platform, where users could freely upload videos, while remaining profitable. Both YouTube and Google Video would end up becoming early success stories in that regard.
It would not be long before these two websites, especially YouTube, became a part of our daily lives. What I am sure will come as a surprise to some readers, is that these two websites share a very much intertwined history, especially when it comes to captioning.
Throughout 2005, Google Video was definitely leading the competition. But once it started to lose ground to YouTube in 2006 it never recovered. Still, Google seems to have believed that there was some hope for Google Video and to that end, they competed to become the better platform by introducing new features. One such feature was, in fact, closed captioning.
(There is supposed to be a Google Trends graph here, but if it somehow fails to load click here)
Search frequency for Google Video and YouTube between 2005 and 2006.
On 19 September support for closed captioning of videos was announced on the Google Video Blog:
Although many of us are responsible for making this possible, it’s particularly meaningful to me because I’m not only an engineer fortunate enough to work on Google Video — I’m also deaf. In some ways this reminds me of when closed-captioning (CC) was first introduced; before that, little on TV made sense and the only movies worth paying for were foreign films, because those were the only ones with subtitles! I now have the same sense of hope that I did then, when you could finally see visible progress and knew for sure that however long it took to perfect things, we really were on the way.
Ken Harrenstien, Google Video blog
For the first time ever, you had a website where you could both host and stream your own videos and captions. At the time, the CC button seemed like a “subtitles on” option, which you could expect from TV or DVDs. Unfortunately, this historical development was quickly overshadowed by Google’s acquisition of YouTube, only three weeks later in October. Nonetheless, it was quite the accomplishment and even YouTube takes this to be the epoch for closed captions on their website, as evidenced by future blogposts and conferences.
Soon after, Google Video began a slow descent into obscurity, as the development team slowly migrated to work on YouTube. Although the website went defunct altogether, we still have some snapshots from this era. The thumbnail for this post is a screenshot of one of the first-ever captioned videos uploaded, titled “Google Video & YouTube Support Closed-Captioning”. Although this photo only shows us the UI, fortunately, the “Me at the zoo” of closed captions survives on YouTube!
The uploader, Dan Greene, who provided us with a lot of insight during my research, later uploaded the video to YouTube in 2009:
Now you might be wondering why anyone would wait so long. The fact of the matter is, it would be another 2 years until YouTube received its own native support for captions. Ken Harrenstien and the rest of the core team working on captions would remain on the Google Video side of things for many more months.
Undaunted, the Google Video team would continue to bring innovations to captioning. While their accomplishments only survive in fragments, a good place to be looking is Google’s own patents, a lot of which have Ken Harrenstien’s name on them. There is US9710553B2 relating to UI design for closed captions and US20140301717A1 relating to support for multiple caption tracks.
There is one patent in particular which is especially relevant to the community contribution feature, US7992183B1: Enabling users to create, to edit and/or to rate online video captions over the web. To date, this is the earliest known proposal for any form of community contribution to closed captions, on Google’s products anyway. Up until this point, if an uploader hadn’t added captions for a video, that was it, you could not do anything about it. There were services like Overstream, which allowed people to caption online videos, but not only were these relatively unknown, you had to actively hunt for a captioned video. This patent describes a method that could have potentially changed that.
This form of community contribution would operate on a rating-based system, multiple users would be able to submit closed captions and viewers would decide on which captions were the best, by rating on a scale from 1 to 5. This differs from the modern system, which enforces one translation per video, and is split into a submission and review phase. In practice, one could imagine this early concept to have worked much faster. I was unable to find any evidence that this idea was ever realized, but at the very least it goes to show that the need for community contribution was there, even back in 2007.
Moving into 2008, Google really started to pick up the pace. On 4 June, we saw YouTube’s answer to captions: Video Annotations!
We’re happy to announce a new way to add interactive commentary to your videos — with Video Annotations. With this feature, you can add background information, create branching (“choose your own adventure” style) stories or add links to any YouTube video, channel, or search results page — at any point in your video.
The YouTube Team, The YouTube Blog
Video annotations were designed to be a lot more versatile than captions, not solely restricted to descriptions of what is being said or happening on the screen. Still, seeing as there still was no native closed captions feature, it would not take long until people figured that they could use these video annotations for subtitling or captioning videos.
Annotations are no longer available on YouTube, so the ones you see here were retrieved via https://invidious.snopyta.org/watch?v=mCg54B69aY4&iv_load_policy=1
And if you are still not convinced that this was a response aimed at the closed captioning, get a load of Google Video’s newest feature that was announced the very next day: Closed Captioning Search! This made it possible to search not only for a particular video but through its contents. An example use case could be searching for a particular talk:
Here’s a nice example – search for [“that’s a tremendous gift”] . Make sure you’ve selected List View, and you should see a video featuring Randy Pausch. Clicking on the “Start playing at search term (50:16)” link will take you to a point slightly before the appearance of that caption.
Ken Harrenstien, Google Video Blog
Believe it or not this feature was eventually brought over to YouTube, though if it works a bit differently.
Sad to say, these features were not much appreciated at the time, as YouTube was suffering from poor layout changes (see comments on the old blog page) and Google Video was suffering from serious stability issues. There was however, a silver lining to all of this.
If you remember visiting YouTube in 2008, you might recall some of the interesting gadgets and features they had introduced for the upcoming presidential election. Let’s talk about a little project called Gaudi.
The premise of Google Video’s caption search, being able to search for things people say, might not very enticing. But what if you wanted to search for things important people, say, politicians have said in the past? So Google’s Speech Research Group developed this tool called Gaudi, which not only allowed you to search through the speeches of presidential candidates but also generated transcripts for relevant news and politics videos as they were uploaded. By bundling speech-to-text
Gaudi is gone now, but it was once available on Google Labs, as a gadget on iGoogle and it was even embedded on YouTube’s YouChoose page offering the latest updates on the presidential campaigns. It even received some media coverage.
If you would like to learn more about the behind the scenes for this one, one of the authors, Michiel Bacchiani, was kind enough to upload a copy of their publication to ICASSP ’09, on his website.
So now we had a form of speech-to-text on the largest video sharing website in the world, no matter how limited. But what to do with this technology? Combining it with video annotations would be too non-trivial. YouTube still didn’t have closed captions either, forcing people to have to use websites like YouTube Subtitler. Perhaps, now was the time to bring the Google Video captioning features over to YouTube!
In the coming months, members from the team that worked on Gaudi, Google Video’s division responsible for closed captions and the existing YouTube team would come together to bring closed captions to YouTube.
Join us next week, to see how all these little pieces would come together to make a whole, truly greater than the sum of its parts.
