How to share audio/video on GNU/Linux

Overview
Requirements
Use tigerVNC and ssh send video through a chain of forwarded ports
- Required programs for video/keypresses
- Example of tigerVNC through a common access point
Use pulseaudio, ffmpeg, ssh, and socat to share audio
- Necessary programs
- Example of audio streaming through a common access point
stream.pl
- All required programs:
- kill everything and start over

Written by:

Robert McIntyre

Overview

This guide will show you how to stream audio, video, and keypresses from one arch-linux computer to another arch-linux computer through a common internet facing server to which both the sender and receiver have ssh access.

You'll find a series of specific examples of how to do the necessary port forwarding for both audio and video. At the end of this page there is a short script (stream.pl) which generalizes the specific examples and provides a solution for streaming which was good enough to satisfy my particular needs.

After reading this you will understand how to forward ports with ssh, remotely control a computer with tigerVNC, transform UDP into TCP and back with socat, and how to use ffmpeg in an unexpected way. Have fun playing games with your friends!

Requirements

Recently, I wanted to watch my friend play a game on his computer. Unfortunately, he lives several thousand miles away, so I needed him to stream the game to me over the internet. I also wanted a more interactive experience where I could also control the game my friend is playing. However, both of us are behind NATs (Network Address Translantors) and don't have public IPs. I hate configuring my router and wanted something that could work anywhere (like a coffee shop). Fortunately for us, we both have access to aurellem.org (the computer hosting this webpage), so it ought to be possible to use aurellem.org as a common point of connection to transfer sound, video and keypresses.

In short my requirements for my streaming solution are:

Work without any additional router configuration.
Transfer video and audio from one computer to another, while not interfering with the experience of the person sending video/audio.
Allow the person reveiving the video audio to also control the game through their keyboard.
The sender and reveiver trust each other, but all connections over the internet must be secure and encrypted.
Low bandwidth.

In all of the examples in this guide, my friend's name is ocsenave, my name is r, and the internet-facing server we are using as a relay point is aurellem.org. ocsenave is the one who wants to play the game, and r is the one who wants to watch the game (and occasionally press some buttons). The game runs on ocsenave's computer and ocsenave streams it to r. Commands that start with ocsenave> are executed ocsenave's computer, and commands that start with r> are executed on r's computer.

Use `tigerVNC` and `ssh` send video through a chain of forwarded ports

Sending video was actually not too difficult. I chose tigerVNC and found it to be quite adequate for my purposes. It automatically compresses video and allows me to reomtely control a computer, meeting all of my streaming requirements except for audio, which tigerVNC does not support.

But how can I actually connect to my friend's computer, which again is behind a NAT? The answer is to forward the ports that tigerVNC uses through aurellem.org using ssh. Then, I can access my friend's computer by pointing tigerVNC at one of my own local ports.

Required programs for video/keypresses

Install tigerVNC and ssh if you don't already have them. On arch-linux you can do this with:

pacman -S openssh tigervnc

Example of tigerVNC through a common access point

`ocsenave` : Create vnc server

First, ocsenave starts a tigerVNC server on his machine. If this is the first time, then he'll be prompted to set up the password that others will need to know to remotely control his computer. Since we're ultimately going to send this through ssh, this password does not have to be very strong.

ocsenave> vncserver -geometry 1440x900 -dpi 96 -alwaysshared \
                    -localhost -rfbport 5000 :1

-geometry 1440x900, -dpi 96: These configure the size of the VNC window and its dpi. If you are having problems with choppy video, you can decrease the size of the vnc window in order to reduce bandwidth.
-localhost: People can only access this server through ports on their local machine. This prevents people from connecting other than through forwarded ports, and is why we don't need to care about the vnc password.
-rfbport 5000: Instructs tigerVNC to listen on TCP port 5000. This is the port ocsenave will need to forward to aurellem.org in order to provide access to his computer to r.
-alwaysshared: allows multiple connections to vncserver. This is necessary to allow both r and ocsenave to both watch the same game!

`ocsenave` : Publish the vnc server's port on the common access point

Now, ocsenave makes his local TCP port 5000 (where tigerVNC is listening) available on aurellem.org using a "reverse ssh tunnel", so that other people with ssh access to aurellem.org can access his tigerVNC program.

ocsenave> ssh -N ocsenave@aurellem.org -R localhost:5001:localhost:5000 &

-N: allows you to send the ssh to the background (using the "&" at the end of the command). ssh will continue to keep the connection open (and the port forwarded) until it is killed.
ocsenave@aurellem.org: Specifies what usser and hostname to forward ports to. ocsenave needs to be able to login without a password to aurellem.org in order for the -N option above to work.
-R localhost:5001:localhost:5000: sets up a "reverse" ssh tunnel with aurellem.org. The first "localhost" means "aurellem.org" and the second "localhost" means ocsenave's computer. Anything sent to aurellem.org's TCP port 5001 will be sent to ocsenave's TCP port 5000, where the tigerVNC server is listening. This allows someont to connect to ocsenave's vnc server by using aurellem.org:5001.

After this is done, ocsenave has successfully provided a gateway to his computer through aurellem.org.

`ocsenave` : log in to the vnc server

While he's waiting for r to connect, ocsenave views his own tigerVNC session locally. Then when r connects, they will both be looking at the same game.

vncviewer localhost:5000

`r` : link the server's port to a local port

Now that ocsenave's connection is available through aurellem.org, r (who also has ssh access to aurellem.org) uses ssh to "forward" one of his local ports to aurellem.org. This completes the connection from one of r's local ports to ocsenave's tigerVNC server.

ssh -N r@aurellem.org -L localhost:5002:localhost:5001 &

-N: as above, allows r to put ssh into the background while keepig the port forwarding active.
r@aurellem.org: r also has an account at aurellem.org and has passwordless login enabled.
-L localhost:5002:localhost:5001: sets up "port forwarding". Anything sent to r's TCP port 5002 will be forwarded to aurellem.org's TCP port 5001. Because of the previous ssh -R command, the information sent to aurellem.org's TCP port 5001 will be further forwarded to ocsenave's TCP port 5000 on which the tigerVNC server is listening.

`r` : connect to `ocsenave's` computer

r runs his own vncclient pointed at the forwarded port:

vncviewer localhost:5002

r enters the password ocsenave configured earlier and now both r and ocsenave can view the same game and both can send keypresses to control the game.

Note that the three different port numbers mentioned (5000,5001,5002) could have just as easily all been 5000 since all these ports are on different computers.

Use `pulseaudio`, `ffmpeg`, `ssh`, and `socat` to share audio

Sharing audio is very similar to video but with more of a "pushing" feel than a "pulling" feel. With video you "invite" someone to share your screen while with audio you "broadcast" your sound and someone else listens. This will be reflected in how we use ssh to set up port forwarding for audio.

The general idea is to use pulseaudio and ffmpeg to gather all the audio playing on the system and send it through a local port, which is similar to the way tigerVNC made video available through a local port. Then we again use a series of ssh commands to tunnel the audio form ocsenave to r. The only difference is that ssh can only forward TCP ports while ffmpeg broadcasts to UDP ports. So we use socat to transform UDP to TCP on ocsenave's end, transport to r's computer with ssh, then again use socat on r's end to transform back from TCP to UDP. Then r can receive the audio stream with ffplay.

Necessary programs

pacman -S socat ffmpeg openssh pulseaudio pavucontrol

Example of audio streaming through a common access point

`ocsenave` : begin capturing audio with `ffmpeg` and `pulseaudio`

ocsenave starts capturing all the audio on his system and streaming it to his local UDP port 6000.

ffmpeg -re -ar 20000 -f alsa -i pulse -c:a mp3 \
   -f rtp rtp://127.0.0.1:6000 > ~/config.sdp &

-re

"realtime" capture – this basically ensures that the audio will be captured and transported at the same rate as the video.

-ar 20000

this is the bitrate of the audio stream. The higher the number is, the "better" the audio will sound. If you start getting popping or static after executing this command, reduce the bitrate until it goes away!

-f alsa

this says that ffmpeg is going to be gathering raw sound data from the system, and to expect it to be in whatever format the system default is.

-i pulse

defines the "input file" for ffmpeg, which is often a file but in this case is whatever pulseaudio wants to send to ffmpeg. You will be able to use pavucontrol to send all system sounds to ffmpeg, just send some sounds, or mute the input entirely.

-c:a mp3

use the mp3 codec to encode audio. This option helps to conserve bandwidth while streaming audio over the internet. For example, if you're transmitting silence via mp3, it hardly takes any bandwidth at all! If you were using an uncompressed format like wav, you would consume quite a bit of bandwidth even while broadcasting silence.

-f rtp

This tells ffmpeg that the output format is actually going to be a streaming broadcast using the rtp protocol.

rtp://127.0.0.1:6000

This tells ffmpeg to broadcast sound to the local UDP port 6000. Something like ffplay will be able to connect to this local UDP port and play or record the sound.

> ~/config.sdp

This directs standard out to the file /home/r/config.sdp. When transporting sound over rtp, it's necessary to include some information describing what sort of sound data is being sent. The config.sdp file will look like this:

SDP:
v=0
o=- 0 0 IN IP4 127.0.0.1
s=No Name
c=IN IP4 127.0.0.1
t=0 0
a=tool:libavformat 56.40.101
m=audio 6000 RTP/AVP 14

The most important part of this file is the line:

m=audio 6000 RTP/AVP 14

at the end of the file. This says to listen at port 6000 for sound.

&

put ffmpeg into the background where it will continue to stream sound until is is killed.

Note that just as r had to get the tigerVNC through some channel, he will also need to have this sdp file on his computer as well.

`ocsenave` : use `socat` to transport UDP datagrams via TCP

Since ssh can only deal with TCP ports, ocsenave uses socat to transport information form UDP 6000 to TCP 6000.

socat UDP4-RECVFROM:6000,fork TCP:127.0.0.1:6000 &

UDP4-RECVFROM:6000,fork: listen to UDP port 6000 without blocking the port from other processes
TCP:127.0.0.1:6000: send everything to TCP port 6000 on ocsenave's computer.

`ocsenave` : use `ssh` to forward audio to the common server

ssh -N ocsenave@aurellem.org -L localhost:6000:localhost:6001 &

Just as with video, this ssh invocation sends everything from the local TCP port 6000 to aurellem.org's TCP port 6001. Note that this is the opposite direction (-L instead of -R) than what we used to share video.

`r` : use `ssh` to receive audio from the common server

ssh -N r@aurellem.org -R localhost:6001:localhost:6000 &

r uses ssh to bind aurellem.org's TCP port 6001 to his local TCP port 6000. Now sound will be streaming to ocsenave's UDP port 6000, through socat to ocsenave's TCP port 6000, through aurellem.org via TCP port 6001, and will come out of r's TCP port 6000. The only thing left to do is to translate back to UDP on r's end and play the audio through the speakers.

`r` : use `socat` to translate back from TCP to UDP

socat tcp4-listen:6000,reuseaddr,fork UDP:localhost:6000 &

tcp4-listen:6000,reuseaddr,fork: get data from TCP port 6000 without blocking…
UDP:localhost:6000: …and forward it all to UDP 6000!

`r` : listen to `ocsenave's` audio

Again r will need the config.sdp file generated by ocsenave's invocation of ffmpeg to make sense of the data that's now coming form r UDP port. This file doesn't change, although if r has chosen to ultimately map audio to a different UDP port than the one in the SDP file, he will have to modify that file to reference the correct port. Once everything's taken care of, r executes:

ffplay -i ~/config.sdp

-i ~/config.sdp: play audio as described by the SDP file at /home/r/config.sdp.

To begin listening to audio on ocesnave's computer.

If r wants to also COPY all the audio he's getting from ocsenave's computer, he can also execute:

ffmpeg -i ~/config.sdp -c:a copy out.mp3

-c:a copy: don't do any processing on the audio but instead copy it directly to a file.
out.mp3: filename where audio will be recorded. Press q to stop recording.

`stream.pl`

This guide has examples to show how you might send and receive audio. I abstracted them into a script, stream.pl that does everything at once so that ocsenave and I can get on with our lives and play our game!

#!/bin/perl

use Getopt::Long;
use Switch;

$mode = shift @ARGV;


# Default Options
switch ($mode){
    case "send"    {$user = "ocsenave"}
    case "receive" {$user = "r"}
}
$server = "aurellem.org";
$local_video_port="5000";
$local_audio_port="6000";
$public_video_port=$local_video_port;
$public_audio_port=$local_audio_port;

GetOptions ("server=s" => \$server,
            "user=s" => \$user,
            "local-audio-port=i"  => \$local_audio_port,
            "public-audio-port=i" => \$public_audio_port,
            "local-video-port=i"  => \$local_video_port,
            "public-video-port=i" => \$public_video_port)           
    or die("Error in command line arguments\n$usage\n");


##################################################
##   Video Sharing                               #
##################################################
if ($mode eq "send"){

    $start_vnc_server=
        "vncserver -geometry 1440x900 -alwaysshared \\
                   -dpi 96 -localhost -rfbport $local_video_port :1 &";
    print("$start_vnc_server\n");
    system($start_vnc_server);

    $publish_vnc_port=
        "ssh -N $user\@$server -R \\
          localhost:$public_video_port:localhost:$local_video_port &";
    print("$publish_vnc_port\n");
    system($publish_vnc_port);
}
elsif ($mode eq "receive"){

    $link_to_server=
        "ssh -N $user\@$server -L \\
           localhost:$local_video_port:localhost:$public_video_port &";
    print("$link_to_server\n");
    system($link_to_server);

    `sleep 4`;
    
    $start_vnc_client=
        "vncviewer localhost:$local_video_port &";
    print("$start_vnc_client\n");
    system($start_vnc_client);

    `sleep 10`;
    
}
    
##################################################
##   Audio Sharing                               #
##################################################
if ($mode eq "send"){
    $udp_broadcast_port = $local_audio_port;
    $tcp_broadcast_port = $local_audio_port;
    $server_relay_port  = $public_audio_port;
    
    $bind_udp_to_tcp = 
        "socat UDP4-RECVFROM:$udp_broadcast_port,fork \\
               TCP:127.0.0.1:$tcp_broadcast_port &";
    system($bind_udp_to_tcp);
    
    $forward_tcp_to_server =
        "ssh -N $user\@$server -L \\
         localhost:$tcp_broadcast_port:localhost:$server_relay_port &";
    system($forward_tcp_to_server);

    $broadcast_sound = 
        "ffmpeg -re -ar 20000 -f alsa -i pulse -c:a mp3 \\
                -f rtp rtp://127.0.0.1:$udp_broadcast_port &";
    system($broadcast_sound);
}

elsif ($mode eq "receive"){
    $receiving_port = $local_audio_port;
    $server_relay_port = $public_audio_port;

    $spd_file = `mktemp`;
    chomp($spd_file);
        
    $SPD = "\\
SDP:
v=0
o=- 0 0 IN IP4 127.0.0.1
s=No Name
c=IN IP4 127.0.0.1
t=0 0
a=tool:libavformat 56.40.101
m=audio $receiving_port RTP/AVP 14
";
    open SPD, ">", $spd_file or die $!;
    print SPD $SPD;

    $connect_to_server=
        "ssh -N $user\@$server -R \\
         localhost:$server_relay_port:localhost:$receiving_port &";
    print("$connect_to_server\n");
    system($connect_to_server);
    
    $bind_tcp_to_udp=
        "socat tcp4-listen:$receiving_port,reuseaddr,fork \\
               UDP:localhost:$receiving_port &";
    print("$bind_tcp_to_udp\n");
    system($bind_tcp_to_udp);

    $play_sound = "ffplay -i $spd_file &";
    print("$play_sound\n");
    system($play_sound);
}

stream.pl: a program for sharing video, audio, and keypresses between friends.

All required programs:

sudo pacman -S base-devel openssh perl perl-switch \
               tigervnc socat ffmpeg pulseaudio pavucontrol \
               wine wine-mono wine_gecko winetricks

kill everything and start over

sudo killall ssh socat ffmpeg ffplay ssh Xvnc

How to share audio/video on GNU/Linux

Table of Contents

aurellem ☉

Overview

Requirements

Use `tigerVNC` and `ssh` send video through a chain of forwarded ports

Required programs for video/keypresses

Example of tigerVNC through a common access point

`ocsenave` : Create vnc server

`ocsenave` : Publish the vnc server's port on the common access point

`ocsenave` : log in to the vnc server

`r` : link the server's port to a local port

`r` : connect to `ocsenave's` computer

Use `pulseaudio`, `ffmpeg`, `ssh`, and `socat` to share audio

Necessary programs

Example of audio streaming through a common access point

`ocsenave` : begin capturing audio with `ffmpeg` and `pulseaudio`

`ocsenave` : use `socat` to transport UDP datagrams via TCP

`ocsenave` : use `ssh` to forward audio to the common server

`r` : use `ssh` to receive audio from the common server

`r` : use `socat` to translate back from TCP to UDP

`r` : listen to `ocsenave's` audio

`stream.pl`

All required programs:

kill everything and start over

How to share audio/video on GNU/Linux

Table of Contents

aurellem ☉

Overview

Requirements

Use tigerVNC and ssh send video through a chain of forwarded ports

Required programs for video/keypresses

Example of tigerVNC through a common access point

ocsenave : Create vnc server

ocsenave : Publish the vnc server's port on the common access point

ocsenave : log in to the vnc server

r : link the server's port to a local port

r : connect to ocsenave's computer

Use pulseaudio, ffmpeg, ssh, and socat to share audio

Necessary programs

Example of audio streaming through a common access point

ocsenave : begin capturing audio with ffmpeg and pulseaudio

ocsenave : use socat to transport UDP datagrams via TCP

ocsenave : use ssh to forward audio to the common server

r : use ssh to receive audio from the common server

r : use socat to translate back from TCP to UDP

r : listen to ocsenave's audio

stream.pl

All required programs:

kill everything and start over

Use `tigerVNC` and `ssh` send video through a chain of forwarded ports

`ocsenave` : Create vnc server

`ocsenave` : Publish the vnc server's port on the common access point

`ocsenave` : log in to the vnc server

`r` : link the server's port to a local port

`r` : connect to `ocsenave's` computer

Use `pulseaudio`, `ffmpeg`, `ssh`, and `socat` to share audio

`ocsenave` : begin capturing audio with `ffmpeg` and `pulseaudio`

`ocsenave` : use `socat` to transport UDP datagrams via TCP

`ocsenave` : use `ssh` to forward audio to the common server

`r` : use `ssh` to receive audio from the common server

`r` : use `socat` to translate back from TCP to UDP

`r` : listen to `ocsenave's` audio

`stream.pl`