AV integration
Introduction
Ever since the earliest incarnations of the Historiana project we envisioned an user interface which allows working with any kind of media. Due to various constraints upto now all interaction is mainly focused on images and text.
Using the concept of the Historiana Card we can introduce other media by implementing a viewer for these content types in the Card component.
This is a first step, both audio and video content open a world of new interactions for both teachers and students. Enabling the viewing of the media is an important step, once implemented we can develop more tooling around these content types.
This document tries to caption all issues and developments related to the addition of AV sources to the Historiana project.
Inventory
Serving video (and to a lesser extend audio) in 2021 can be a complex matter. Back in the early days of dial-up internet it was complicated because the lack of sufficient bandwidth to the client. Now the main area of complexity is the wide variety of different clients.
Desktop computers are significantly different from mobile phones but both are completely useable as a device to consume video. Bandwidth could be a concern but only for high resolution materials on slow (mobile) devices. Apart from the desktop/mobile split there is also an issue with screen-estate. An modern iPad has more pixels than an aged desktop computer.
AV content can be self-contained in a .mp4 or similar video file format or it could be posted online on one of the video-platforms where YouTube and Vimeo are leading but there are numerous others.
In the early days the growth of YouTube was mostly due to the ease of use and the free storage of data. In the mean time YouTube has developed into a leading platform because it deals with many of the intresic details of serving video.
One can place a video file on a server and think its done, these sites present the video via a dedicated player on a webpage, for instance with JW-player. This is the old school way of doing things which were created because browsers could not deal with video content at that time.
Development of support for video and audio was started with the establishment of the HTML5 standard and most browsers now do have native support for playing AV content.
However they also assume certain functionality from the server side, they expect the ability to stream the content and a webserver serving .mp4 files can't do such things. Platforms as Vimeo and YouTube however do.
Within the scope of Historiana another aspect is the longevity of the content, we need to guarantee the availibilty of the content because it is used in our content, the last thing we want is an eActivity with a missing external video.
Challenge
We want to support both audio and video, in the common formats supported by modern browsers. External sources like Vimeo and YouTube can be used as a source for materials but we need to make sure that important content is kept locally.
We need a solution which supports modern browsers regardless of the device. Thus we want a player which is supporting Desktop, Tablet and Mobile.
In order to test these things within the scope of the harsh reality of cross-platform and cross-browser support we created a test in which we experiment with several players and platforms: https://dev.historiana.eu/Testpage
Streaming
Although it seems temping to just store an .mp4 somewhere in the cloud and serve it as-is this would harm the user experience (UX) on all platforms as most browsers nowadays except a streaming service while playing AV content. This allows the user to watch the content quickly as the viewing can start even while the download is not finished yet. Serving just MP4s would imply download and then view on many platforms. More on this can be found at u3m8 - Live Streaming.
Testing
On the testpage we test embedding various players and various methods of hosting. To enable us to easily host video within the project we are testing with mux.com which offers a video hosting platform with an easy to use API. They explicitely state that they offer non-profit discounts https://mux.com/pricing/
History of MUX : https://en.wikipedia.org/wiki/Mux_(company)
We are testing:
- the standard HTML5 tag to support video, https://developer.mozilla.org/en-US/docs/Web/HTML/Element/video
- the Quasar equivalent; https://quasar.dev/vue-components/video#qvideo-api
- the Quasar recommended advanced player; https://github.com/quasarframework/quasar-ui-qmediaplaye
We test with both files hosted on MUX as mp4 and m3u8 (adaptive playlist), Europeana hosted .mp4 and YouTube.
| type | file | ios | ipad | android | safari | chrome | firefox |
|---|---|---|---|---|---|---|---|
| video | mp4 | ok | ok | ok | ok | ok | |
| q-video | mp4 | ok | ok | ok | ok | ok | |
| q-mediaplayer | mp4 | ok | ok | ok | ok | ok | |
| video | m3u8 | nok | ok | ok | via hlsjs ok | via hlsjs ok | |
| q-video | m3u8 | ok | ok | ok | download | download | |
| q-mediaplayer | m3u8 | ok | ok | ok | no play | no play |
There is an issue with the pause/play button with the q-media-player as visible in this image
Development
Development was done with these design parameters in mind:
- Europeana content is leading for the current development
- Primary goal is video; audio only is just video without moving images
- Other external content should be viable with same solution
- Streaming is really needed
- Build reusable elements as the current developments are leading to more development in the future.
With these in mind we started work. One of the major issues was the research for the streaming of video. Platforms like YouTube and Vimeo are streamlined for theses tasks but are bad to integrate, the main reason for their existence is generating revenue from viewers, either via advertisements or membership.
Easy integration into an educational platform is not one of their priorities and hence they offer little to no support for programmers to embed their services. The best one can expect is an <iframe> based integration into a page.
Creating the necessary services to support streaming video is a major task by itself and clearly outside the scope of this project.
There are various open source solutions towards video streaming, some of them are geared towards content makers who want their own solutions. https://opensource.com/article/19/1/basic-live-video-streaming-server while others are more hosting minded like https://www.wowza.com/products/streaming-engine but all are based around the concept of hosting (and therefore running) your own services.
While running a web-application server is not a big task and we run these -self-hosted- as part of the Historiana Project adding a video layer to this stack was something we did not want to do at this moment.
During our research we stumbled upon https://mux.com/ ; a well-funded startup which runs video hosting as a service with the programmers perspective in mind. They offer the automatic conversions needed to stream video in various formats to a wide range of clients. In addition they also have services to generate thumbnails from video and support for captions.
Currently the only main thing lacking is the automated generation of captions (audio transcribing) but for the embedding of AV materials in Historiana this is not a big issue right now. There are several external services who we could use to generate captions, the MUX API does support the use of these captions.
An overview of all services provided is at https://docs.mux.com/
By using the MUX provided services we can concentrate on how we want to integrate the concepts of audio and video within the Historiana context. One of the early conclusions was that it needs to be an integral part of both the standard content sections and the Apps section where applications can be created using the Historiana content.
To integrate AV in Historiana we decided to start with these:
- import video from Europeana into MySources
- allow upload of video in MySources
- view AV content in the Historiana Card
- extend Apps to allow usage of video
- add functionality to the Historiana API to support all above, using MUX
Adding to Historiana
In the MySources section of Historiana members can find and select Europeana sources and/or upload their own sources. Before the AV integration the uploading was limited to images only. To enable the use of AV materials we extended both the user-interface and back-end to enable the selection / uploading of AV content.
The integration for Europeana content is currently limited to the Partners only, anyone with an account can upload an existing video. The latter will be processed at mux.com and stored in a card. The Europeana integration is done by fetching the video from the original provider, process it at mux and store the metadata to view it in a card.
Once a video is imported as a source it can be used in eActivities as well as in other places.
Additional meta-data can be entered via the local edit function
And it can be used in an eActivity:
You can view this in action by clicking the video (hosted at mux.com)
With these first steps we can enable our members and partners using video in their content while also using these materials in the new Narratives.
The API developed is very centric around the workflows in Historiana and uses MUX as a back-end; when needed in the future we can simply add other video-processing backends or replace the MUX one if that need arises.
Lessons learned
- video streaming is an expertise on its own, use a 3rd party.
- streaming player is crossplatform without too much hassle.
- watching and adding video is one thing, doing stuff with it another.
Initially we had some thoughs, documented in this first draft ; the one main thing we did not answer yet is if our members have need for an editor like the Noterik Video Editor, this question can be carried forward to the research after this project.
Our main focus was enabling video as a source, this has been completed.
Further development
- extend the UI to enabel the user to see a card contains video/audio content.
- preview the video within the context of the card (click on thumb plays video).
- work with feedback and results of the reseearch output of the "Watching video like an Historian" project.
- review old notes, take unanswered issues forward.
Last update: Februari 21 2022