HowTo

This section contains quick guides and HowTos to building and configuring Feng and the other members of the LSCube streaming suite

Building from sources

Standard instruction to use git and autotools to build netembryo, feng, libnemesi, felix.

Accessing the live sources

Our current scm is git, you may access it using the web interface or it's native client. A simple tutorial is available on the website. Every repository is named as the program with a “.git” postponed, e.g. the feng repository is feng.git To fetch the sources the first time is enough to issue

git clone git://git.lscube.org/feng.git

in order to create the repository with the already checked out source. To update the local cloned repository is sufficent to enter the directory and issue

git pull

Gitweb provides also useful autosnapshots in case git isn't available.

Using autotools

All our projects are using GNU autotools. You should install at least automake 1.8.x autoconf 2.5× and libtool 1.5.x (on Debian based distributions install autoconf, automake and libltdl3-dev packages). The following code will generate the correct Makefiles

cd ${project}
autoreconf -v -i

Install dependencies

Some of our projects have external dependencies/recommendations. Look at the INSTALL file provided with the source code.

Compiling

./configure ${options}
make
make install

Will build and install the project.

Summary

To sum up the operations the following lines will install any project from LScube, please refer to the README and INSTALL files to have informations about the project specific dependences.

git clone git://git.lscube.org/${projectname}.git
cd ${projectname}
autoreconf -v -i
./configure
make
su -c make install

Distribution specific instructions

Currently there are live git ebuild for gentoo available here

Deploy Feng

Getting Started

Assuming you already installed feng using the live sources or the gentoo packages you should have at least the following files installed:

/etc/feng.conf
/usr/share/man/man1/feng.1.bz2
/usr/bin/feng

NOTE: the initscripts are provided ONLY by the Gentoo ebuild since they are Gentoo specific, feng can be launched as service using start-stop-daemon.

As you can see feng installation is quite compact, you just have the main executable, the manpage and a main configuration file.

## lighttpd like configuration
var.basedir  = "/var/feng"
var.logdir   = "/var/log/feng"
var.statedir = "/var/lib/feng"

# server.bind = "localhost"

$SERVER["socket"] == "localhost:5554" {
    sctp.protocol       = "enable"
    sctp.max_streams    = 16
}

server.port          = 554
server.username      = "feng"
server.groupname     = "feng"
server.max-connections = 100
server.document-root = var.basedir + "/avroot"
server.errorlog      = var.logdir  + "/error.log"

The configuration files contains the informations needed to have the server started:

Additional entries are available for optional features

Starting Feng

the quickest way to start feng is the following

feng -v

This way feng will try to read the default configuration file and bind the ports accordingly. The program outputs directly on the console in order to facilitate debug.

Preparing contents

Feng can parse streams contained in MOV, NUT, avi and the other containers libavformat can understand. In order to provide a content is enough to copy (or link) the file container in the root directory pointed by the config file. Any request for a file will make feng look for it, parse it and stream it.

Additional features like stream composition and stream aggregation are available using the .sd and the .ds configuration files. Those features aren't yet stable and the syntax is pending a major overhaul.

Streaming vorbis and theora is supported and compatibility with gstreamer had been verified, sadly it is the only available alternate implementation beside our own.

Multicast support

Multicast support for live feeds is currently in development as a standard built in. Again this feature requires some manual configuration using the present sd syntax that is pending a complete overhaul.

Currently using multicasts requires configuring a .sd metademuxer To enable multicast it is just needed to prepend “multicast ip” option to the metademuxer. Also it is needed an “pilot client” which must start the stream (A simple dump_stream from libNemesi is enough) and which will be the reference for the begin and end of the transmission. When the “pilot client” will start the multicast will be enabled and when the “pilot client” terminates also the multicast will be terminated.

A simple example for the metademuxer configuration is the following:

multicast 224.124.0.1
stream
        file_name audio2.mp3
        priority 1
        payload_type 14
        clock_rate 90000
        encoding_name MPA
        port 5502
stream_end

This will enable multicast stream on every system on the same net ( For more informations about multicast reserved ips look at RFC3171 and the IANA )

Live Streaming with Flux

Introduction

Live streaming is a quite complex environment, this guide will try to introduce the user to Live streaming using the LSCube suite in a simple and pragmatic way.
The guide consists of three parts:

Live streaming Audio Only Content

Live streaming inside Feng is currently performed using posix message queues.

To perform live Streaming first of all is necessary to expose the shared resources to the world with a .sd files
For example we might want to stream a simple AUDIO content by itself, then we create a simple .sd file and place it inside the avroot of feng, let's call it “my_radio.sd”

my_radio.sd:

stream
        file_name mq:///audio
        encoding_name MPA
stream_end

Now we have told to feng to reproduce the content of the shared memory named audio.rtp when my_radio.sd is requested.
With this setup we are asking feng to play the content of the message queue named /audio as a live source specifying that the content of the stream has been encoded with MPA (mpeg audio).
If you are using dynamic payload, H264 or AAC for example, you must specify by yourself the payload_type, clock_rate and fmtp for the stream that you are producing. Feng is able to autodetect live data only for static payload types and has been tested only with MPA and MPV.

Now we must actually put the content of our stream inside the message queue.
This can be performed using Flux. Flux has the mq output filter that can write RTP packets inside a define message queue.
We tell Flux to get packets arriving from port 2900 on UDP protocol and copy them without changing them into the /audio posix message queue which is the same from which we told feng to serve the content (remember to check that feng has access rights for the posix message queue) by launching the following command:

flux -c udp://2900*copy*mq:///audio

notice: you may have a queue limit too low set, in order to raise it you may use either ulimit -q unlimited or set it through other specific ways (e.g though pam_limits).

Flux will also create the message queue if it doesn't exist and will destroy it after closing Flux (which can be terminated by simply pressing CTRL+C)

Now we have configured everything that is necessary on our server to expose the RTSP resource and serve it, we just need to actually produce the content to serve. This can be performed with any tool able to generate RTP packets (like ffmpeg or VLC) and send them through the UDP protocol in our case.
Let's try to do it using VLC:

* Run VLC
* Choose the Media->Streaming menu
* Choose any Audio File to be streamed
* Flag the “RTP” option from the “Outputs” box
* Enter the IP address of the machine where Flux and Feng are running in the first enabled “Address” entry field
* Set as “Audio Port” the value 2900 which is the we told to Flux to read from
* Choose the “Audio Codec” Tab
* Flag the “Audio” check and let any option to its default value (MPEG Audio, 128kb/s bitrate and 2channels)
* Click the Stream button and just let VLC produce the content

Now we can listen to our Audio stream simply by connecting with any RTSP+RTP player (like VLC itself) to our stream.
Just open the url rtsp://server_address/my_radio.sd inside VLC and you should be listening to your song Live.

To really make this a live stream we would need to rewrite the timestamps of the stream to serve it as it has been produced NOW, this can be performed by simply launching flux with the live parser instead of the copy one.
Having Flux run with the live parser will also permit to us to change on the fly the audio file that we are streaming without having to restart Flux, Feng or the client itself as the new audio file will be presented as it was still a piece of the previous one.

Live Streaming Audio+Video Content

Streaming an audio and video content is just like streaming two separated audio and video content.
The only thing that will change is that we have to tell feng to serve them togheter.
How to serve the live content is specified inside the .sd file as we saw before, so to serve an audio and video stream paired we just have to specify both of them inside our .sd file

my_tv.sd:

stream
        file_name mq:///audio
        encoding_name MPA
stream_end
stream
        file_name mq:///video
        encoding_name MPV
stream_end

Also we have to tell Flux to inject both the audio and video streams inside the shared memories audio.rtp and video.rtp this can be performed by configuring two chains:

flux -c udp://2900*live*mq:///audio -c udp://2902*live*mq:///video

Note also that we changed the parser from copy to live.
This is to ask to Flux to sync the timestamps of the two streams.
Flux will sync the timestamps of every stream inside chains generated from the same process.

After feeding your stream to ports 2900 for audio packets and 2902 for video packets from VLC or FFMPEG you will be able to view your video from rtsp://server_address/my_tv.sd

Reliable Live Streaming

Until now all the examples used UDP protocol to send the stream data to the server as this is how the RTP producers like ffmpeg and VLC now send RTP packets. But UDP protocol isn't the best option to serve the main stream from which every stream of the users will be duplicated, if we lost a packet or if we have misordering every user will suffer the same problem even if his communication channel is perfect.

To solve this problem Flux introduced support for transmission of RTP packets over TCP.
Using TCP we will introduce a bit of latency but we will have the guarantee that our packets will arrive and will arrive in the right order.

We will solve this problem in a quite straightforward way
We will setup a Flux on the server to receive the TCP packets and serve them to Feng
We will just tell our producer to send the packets to localhost to remove network problems
We will setup a Flux to receive those packets and resend them over TCP

First of all we must configure the Flux that will receive the packets from TCP and will give them to feng (remember to run this on the server where feng is running)

flux -c tcp://1900*copy*mq:///audio -c tcp://1902*copy*mq:///video

This instance of Flux will just wait for TCP connections from another flux on port 1900 and 1902 and will inject every packet that will receive inside Feng.
Then we will setup our local Flux (remember to also change your producer to send packets to localhost instead of directly to the server)

flux -c udp://2900*live*tcp://server_address:1900 -c udp://2902*live*tcp://server_address:1902

This Flux will try to connect to the other Flux on the server with the TCP protocol, will receive packets from the local source, will sync the timestamps as required as it is a live stream and will resend them to the remote Flux.

This way we will have only reliable network channels between our procuder and the Feng server as we will send over UDP to localhost which is supposed to be problems free and we will send over TCP to the remote server having the TCP protocol to mitigate network problems and Flux to automatically recover from timeouts and link problems.

Deploy Theater

Deploying Theater

Theater is based on the Turbogears framework and SQLObject has its ORM. So first of all you need to install this two.
To get the Turbogears framework you can simply download and install it by running the tgsetup.py script as root.
After having installed Turbogears you have to install SQLObject by running easy_install ‘SQLObject=0.9.7’
Then you can simply setup Theater itself.

By default Theater will run on a sqlite database ( to change this please read the documentation at Turbogears Page ) you can initialize the database by running two instructions.
The first will create the schema and the second one will fill it with some basic data like an admin account (Remember to change the admin password inside init_database.py before running it)
First of all run: tg-admin sql create to setup the schema, then run tg-admin shell to open a python shell with access to the created database.
From this shell import the init_database module which will perform the initialization process

amol@quasar ~/web/theater $ tg-admin sql create
Using database URI sqlite:///home/amol/wrk/theater/devdata.sqlite
amol@quasar ~/web/theater $ tg-admin shell
Python 2.5.2 (r252:60911, Aug  4 2008, 13:06:43)
>>> import init_database
Creating Groups and default admin user
2008-12-24 12:16:47,047 turbogears.identity.soprovider INFO Succesfully loaded "theater.model.User"
2008-12-24 12:16:47,048 turbogears.identity.soprovider INFO Succesfully loaded "theater.model.Group"
2008-12-24 12:16:47,048 turbogears.identity.soprovider INFO Succesfully loaded "theater.model.Permission"
2008-12-24 12:16:47,048 turbogears.identity.soprovider INFO Succesfully loaded "turbogears.identity.soprovider.TG_VisitIdentity"
DONE
>>> ^D
Do you wish to commit your database changes? [yes]y

Now simply run start-theater.py from the theater directory to have it serve the 8080 port.

Theater Configuration Tuning

Every option in theater is configured inside the dev.cfg file for development mode and prod.cfg file for production mode.

Changing Feng configuration file location

The Feng configuration file can be changed from the theater configuration file.

feng_conf = "/etc/feng.conf"

Running Theater on a different port

Inside Theater configuration file there is an option for the port to serve. Simply uncomment it and set it to the required value

# server.socket_port=8080

Setting Theater in Production mode

Production mode is faster but has less powerfull error detection, while in production mode the exceptions traceback won't be printed anymore and a simple 500 error will be generated.

To switch Theater in production mode copy the dev.cfg file as prod.cfg, change server.environment=“development” line to server.environment=“production” and then remove the setup.py file. (Remember to delete it as it is the one which will make theater read prod.cfg instead of dev.cfg as configuration file)

Now restart Theater to have it in production mode

Setting Theater behind an Apache Proxy

To have Theater run behind an apache proxy you have to configure a Virtual Host to serve the requested uri. Then add the Proxy options:

ProxyPreserveHost on
ProxyPass / http://localhost:8080/
ProxyPassReverse / http://localhost:8080/

Setting Theater to serve a different url instead of the root of a vhost

As before you have to setup the Proxy to redirect the requested url to Theater.
For example if you want to serve the /media directory

ProxyPreserveHost on
ProxyPass /media http://localhost:8080/
ProxyPassReverse /media http://localhost:8080/

Now you have to tell to Theater that it is running on a different URI
To do this you have to change the theater configuration file by adding thos two lines:

base_url_filter.on = True
base_url_filter.base_url = "http://lscube.top-ix.org"

The second option has to point to the base_url of your domain (like http://lscube.top-ix.org in this case)
Then you have to tell to Theater which directory it has to serve (which is /media in this case):

server.webpath="/media"

Remember that Theater will now serve from the feng avroot only the content inside its webpath
So if you choose to have Theater serve the /media path every video file will be uploaded and looked up inside a /media subdirectory inside the feng avroot. If you have /var/feng/avroot as your avroot only files from /var/feng/avroot/media will be served. This directory has to exist.

Theater Guide

Theater Main Window

The main page of Theater displays the content of the feng root as a navigable tree.
If logged in options to upload new content, to create new live streams and to edit the feng configuration will appear.
Also under each directory or media content buttons to delete the content or the directory appear (pay attention that this will also remove them fron your feng avroot)
Under media content the edit button will also appear, which permits to update metadata for the selected content



Feng Control Panel

Inside feng control panel it is possible to change the main options of feng (saving them will restart feng) and also to check the status of the feng process to see if it is currently RUNNING, STOPPED or ZOMBIE.
Near the status report it is available a link to force feng restart.
Pay attention that the server host will be used by theater to generate RTSP links to serve the resources

Remember that Theater will work as an angel for feng, restarting it if it crashes or if it isn't running when any content is requested from the web interface



Video on Demand Upload

Inside upload section it is possible to create a new Video On Demand resource
You can choose:



Video on Demand View

Inside the view page users can see the content itself.
Title, author and description will be displayed if available and an instance of VLC will be embedded with VLCObject and VLCControls to permit to see and control the video.



Video on Demand Editing

When logged in a pen icon will appear near the name of each resource.
Clicking on this an edit section will open up to update informations about the title, author and description of the resource



Live Streaming Configuration

Live streaming configuration is a complex thing but Theater hides all the complexity behind a simple form.
You will just have to choose a:



Live Streaming View

Inside the conference view you will have:

For logged in users also Moderate Wall and Edit Timeline actions will be available to add events to the timeline or to accept/reject messages sent by users

At the url /wall/wall/conferece_id a bigger version of the wall is available to permit to show it to people at the conference on a big monitor.



Live Streaming Messages Moderation

Messages moderation is quite easy. You will se a list of past and present messages, by default each message is with Accepted state false, by simply clicking on a message row the state will be switched to true and the message will be displayed.
The messages list will update by itself, but you can force list regeneration by pressing Update link



Live Streaming Events Registration

Conference Events registration inside the timeline is as simple as messages moderation
A list with each past and future event for the conference will be displayed, by simply clicking on any row the edit panel for the event will appear.
To add new events you can click the Add New Event link on the top right

The Edit event panel will slide in over the event list
It will permit to set:

A Delete Event link will also be available to delete the currently edited event. The Close link will close the edit box without applying any change