Add RTMPS relay support

[mannyamorim]

This is an attempt to add support for the relay module to directly connect to RTMPS targets. This is valuable for security and is required for certain streaming services (Facebook). Please test if you can, feedback is welcome 👍

Notes

• Implemented a number of new SSL directives which are patterned after the ngx_http_proxy_module using the prefix rtmp_relay_. The directives follow the same syntax as the proxy module. Details are added to the doc folder.
• Added two new parameters for relay targets to allow you to override certain behaviors. This is also documented.
• A lot of code is patterned after the ngx_http_proxy_module and the ngx_http_upstream_module.
• All of the new code is conditionally compiled using a new macro NGX_RTMP_SSL which is an alias for NGX_OPENSSL.
• The only change to existing code is moving the ngx_rtmp_relay_app_conf_t to the header file.
• Also included one bugfix discovered while testing, which is a segfault if the initial push fails. Details in the commit description for the fix.
• Added support for the notify module to use RTMPS when receiving a 3xx response in an on_publish callback.
• Added support for the notify module to resolve a hostname using DNS asynchronously when receiving a 3xx response in an on_publish callback.

Sample Configuration

I provide a sample configuration file below to demonstrate some of the possible options. The purpose of this config is a multi streaming server with one application that will push to YouTube and Facebook live. You will need to replace the xxxx with your streaming key.

load_module modules/ngx_rtmp_module.so;

worker_processes 1;

events {
    worker_connections 128;
}

http {
    server {
        listen 8080;

        location /facebook {
            rewrite ^.*$ rtmps://live-api-s.facebook.com/rtmp/****? permanent;
        }

        location /youtube {
            rewrite ^.*$ rtmps://a.rtmps.youtube.com/live2/****? permanent;
        }
    }
}

rtmp {
    resolver 1.1.1.1 8.8.8.8 ipv6=off;
    resolver_timeout 30s;

    rtmp_relay_ssl_protocols TLSv1.2 TLSv1.3;
    rtmp_relay_ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
    rtmp_relay_ssl_server_name on;
    rtmp_relay_ssl_verify on;
    rtmp_relay_ssl_verify_depth 5;
    rtmp_relay_ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt;

    notify_relay_redirect on;
    drop_idle_publisher 10s;

    server {
        listen 1935;

        application test {
            live on;
            drop_idle_publisher 10s;

            push rtmps://a.rtmps.youtube.com/live2/xxxx;
            push rtmps://live-api-s.facebook.com/rtmp/xxxx;
        }

        application test2 {
            live on;

            push rtmp://127.0.0.1/facebook;
            push rtmp://127.0.0.1/youtube;
        }

        application facebook {
            live on;

            on_publish http://127.0.0.1:8080/facebook;
        }

        application youtube {
            live on;

            on_publish http://127.0.0.1:8080/youtube;
        }
    }
}

cc: #165 arut#1619 arut#747 arut#1605 arut#1408 arut#1587 arut#1604 arut#1397 arut#457 arut#232 arut#191 arut#68

[yildizsc]

@mannyamorim This is great work. When this would be merged into the main repository?

[mannyamorim]

@mannyamorim This is great work. When this would be merged into the main repository?

Thanks @yildizsc. Merging is up to @sergey-dryabzhinsky.

[RainNameless]

Does it support tiktok?
rtmps://push-rtmp-l11-va01.tiktokcdn.com:443/game

[mannyamorim]

@RainNameless I've verified the code against Youtube and Facebook, but it should support any RTMPS target. Please test and let me know if there are any issues with Tiktok.

[Be1erafon]

@mannyamorim Tiktok and instagram works! Thanks for adding rtmps!

[Be1erafon]

Telegram doesn't work. Any ideas what is the reason?

server {
    listen 1935;
    application telegram {
        live on;
        push rtmps://dc4-1.rtmp.t.me/s/1234567890:xxxxxxxxxxxxxxxxx;  
    }
}

relay: SSL certificate does not match "dc4-1.rtmp.t.me/s/1234567890", client: dc4-1.rtmp.t.me/s/1234567890:xxxxxxxxxxxxxxxxx, server: ngx-relay

[mannyamorim]

@Be1erafon it looks like the SSL name is being calculated incorrectly. This is used for the TLS SNI field and for certificate hostname validation hence the failure to validate. This is happening since there is a colon in the URL that is not denoting a port number. Apparently this is legal, so I will need to tweak the code a bit to handle these cases correctly. I anticipate that I can have a fix available sometime tomorrow.

As a stopgap measure you could set: rtmp_relay_ssl_verify off; to disable any certificate validations.

[mannyamorim]

@Be1erafon I've pushed a commit which should fix the issue you were experiencing. Please test again and let me know the results.

[Be1erafon]

@mannyamorim Connection is working.
The video stream in the telegram does not go. According to the statistics, there is a connection, the data is being sent.

Doesn't work

application telegram {
    live on;
    push rtmps://dc4-1.rtmp.t.me/s/1234567890:xxxxxxxxxxxxxxxxx; 
}

Works via ffmpeg

application telegram-ffmpeg {
    live on;
    exec_push ffmpeg -re 
        -i rtmp://127.0.0.1/telegram-ffmpeg 
        -c:v copy -c:a copy 
        -f flv rtmps://dc4-1.rtmp.t.me/s/1234567890:xxxxxxxxxxxxxxxxx;
}

Looks like Telegram has changed something. Worked with nginx a few days ago.
ps: youtube, tiktok, instagram by rtmps works.

[mannyamorim]

@Be1erafon thanks for letting me know. I'll try to work out what the difference is between the NGINX module and the FFMPEG implementation. Will post back once I have a better understanding of this problem.

[Rosstarz]

this is really cool.

could you give me a quick guide on how to install nginx with your version of nginx-rtmp module? I tried downloading your repo ssl branch

git clone -b ssl https://github.com/mannyamorim/nginx-rtmp-module.git

and then following nginx guide https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-open-source/#sources and then do

./configure --add-module=/path/to/nginx-rtmp-module make make install

but when I run nginx with your configuration I get

nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_rtmp_module.so" failed (/usr/share/nginx/modules/ngx_rtmp_module.so: cannot open shared object file: No such file or directory) in /etc/nginx/nginx.conf:6

any ideas?

[sergey-dryabzhinsky]

You should build dynamic module:

./configure --add-dynamic-module=/path/to/nginx-rtmp-module
make make install

[mannyamorim]

@Rosstarz you can read a bit about dynamic modules in NGINX here: https://www.nginx.com/resources/wiki/extending/converting/.

You can use the commands that @sergey-dryabzhinsky posted to build as a dynamic module. If you would like to use it as a static module instead (the way you were originally trying), you will just need to remove the line at the top of the sample config file load_module modules/ngx_rtmp_module.so;. When compiled as a static module the directives will be available without the need to load any additional code.

[yildizsc]

Hi @mannyamorim,

Thanks again for this great afford. This setup is actually working for AWS IVS. But I need to make rtmp_relay_ssl_verify off to make it work while using self-signed SSL.

Here is my configuration:

  rtmp_relay_ssl_protocols TLSv1.2 TLSv1.3;
  rtmp_relay_ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
  rtmp_relay_ssl_server_name on;
  rtmp_relay_ssl_verify off;
  rtmp_relay_ssl_verify_depth 5;
  rtmp_relay_ssl_trusted_certificate /certs/domain.crt;

  notify_relay_redirect on;

And here is the code which proceses self-signed SSL:

mkdir /certs && cd /certs && openssl req \
  -newkey rsa:2048 -nodes -keyout domain.key -subj "/C=FR/O=krkr/OU=Domain Control Validated/CN=*.domain.com" \
  -x509 -days 365 -out domain.crt

Do I need an authoritative SSL?

[mannyamorim]

@yildizsc from Nginx's point of view there is no difference between authoritative SSL and self-signed certificates. Both should work just fine.

In this case, I believe you need to create a CA certificate (self-signed of course) first and then use it to sign a client certificate for your application. The CA certificate would then be what you pass into the rtmp_relay_ssl_trusted_certificate directive.

If you're still having trouble, please post the error logs that you get. If nothing descriptive, please add the debug option to the error_log directive. I'll need to see the specific error message to understand why the SSL verification is not working.

[yildizsc]

@mannyamorim

2022/05/22 06:24:23 [error] 14#14: *10 relay: SSL certificate verify error: (20:unable to get local issuer certificate), client: somertmp/live/token, server: ngx-relay
2022/05/22 06:24:23 [error] 14#14: *10 relay: SSL certificate verify error: (20:unable to get local issuer certificate), client: somertmp/live/token, server: ngx-relay

This is what I get with debug mode.

Btw, could you elaborate on creating the certificate first? Does it matter? I could get the point.

[mannyamorim]

@yildizsc please ignore my earlier comments about needing to have a CA certificate. It appears that this is not necessary.

The error message "unable to get local issuer certificate" didn't turn up anything terribly useful. It just seems to be a generic message that the openssl library can't load your certificate. I just ran a quick test with a single self signed certificate and it seems to be working fine. If you try out this setup, hopefully you can work towards your setup and see where the difference is that's causing the issue. Also, could you post your full Nginx config file (with anything sensitive removed/replace with dummy values), hopefully that can help to shed some more light on the problem.

Test setup was Nginx relaying to an Stunnel instance with a self signed certificate with certificate generated with the following command:

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -sha256 -days 365 -nodes -subj '/CN=localhost'

Stunnel conf:

[RTMP IN Server]
accept = 127.0.0.1:1937
connect = a.rtmp.youtube.com:1935
cert = /etc/stunnel/cert.pem
key = /etc/stunnel/key.pem

Nginx conf:

rtmp {
	rtmp_relay_ssl_protocols TLSv1.2 TLSv1.3;
	rtmp_relay_ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
	rtmp_relay_ssl_server_name on;
	rtmp_relay_ssl_verify on;
	rtmp_relay_ssl_verify_depth 5;
	rtmp_relay_ssl_trusted_certificate /etc/stunnel/cert.pem;

	server {
		access_log /nginx/access.log;
		listen 1935;

		application test3 {
			live on;
			push rtmps://localhost:1937/live2/xxxx;
		}
	}
}

Please note that the CN (Common Name) of the certificate needs to match the name used in the push command in Nginx. Alternatively the SAN field could be used as well. In this case I've set the CN to localhost but you can use any domain.

[kuznetcoff777]

@Be1erafon thanks for letting me know. I'll try to work out what the difference is between the NGINX module and the FFMPEG implementation. Will post back once I have a better understanding of this problem.

Hi @mannyamorim, if any success with this? Faced the same issue with telegram...
When use push rtmps - no erros in debug, just see as stream goes there (tcpdump also prove it), but telegram says he does not see it.

[mannyamorim]

@Be1erafon thanks for letting me know. I'll try to work out what the difference is between the NGINX module and the FFMPEG implementation. Will post back once I have a better understanding of this problem.

Hi @mannyamorim, if any success with this? Faced the same issue with telegram... When use push rtmps - no erros in debug, just see as stream goes there (tcpdump also prove it), but telegram says he does not see it.

Hi @kuznetcoff777, sorry for the delay. I haven't been able to figure out this mystery yet. I ran a test with the NGINX stream module to see if it is an issue with the SSL code in this PR. I used the following configuration:

worker_processes  1;

error_log  xxxxx;

events {
	worker_connections  1024;
}

stream {
	server {
		listen 5000;

		proxy_pass dc1-1.rtmp.t.me:443;
		proxy_ssl on;
		proxy_ssl_protocols TLSv1.2 TLSv1.3;
		proxy_ssl_server_name on;
		proxy_ssl_verify on;
		proxy_ssl_verify_depth 5;
		proxy_ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt;
	}
}

rtmp {
	server {
        	access_log xxxxx;

		listen 1935;

		application test {
			live on;
			push rtmp://127.0.0.1:5000/s/xxxxx;
		}
	}
}

Essentially there is a single push command that pushes to a local endpoint which is then proxied to the SSL destination at Telegram using the NGINX SSL stream module. This failed with the same symptoms as with this PR. The debug error log shows that the RTMP session is operating normally but nothing is displayed in Telegram. This looks to be an issue in the main RTMP module and not with this PR. The guts of the RTMP protocol is really outside of my knowledge, so I don't think that I will be able to debug further.

I have logged this issue as #350. At this point I don't consider it an issue with this PR and likely won't be looking into it any further. Please let me know if you have a different opinion or further information.

[bertin0]

I see that you have added a dockerfile in your fork. Could you provide a basic tutorial of how to run it?

[mannyamorim]

I see that you have added a dockerfile in your fork. Could you provide a basic tutorial of how to run it?

@bertin0 the Dockerfile that I have added to my fork is on a different branch, and I am not planning on submitting a PR for it, so not really relevant here. That being said, the usage is very simple. Basically, the container is just a standard nginx docker container plus the rtmp module added as a dynamic module. You can refer to the documentation for the nginx docker container here: https://hub.docker.com/_/nginx. I would recommend just mounting the config file directly. For the configuration file, you can refer to the documentation in the PR.

[haerong22]

@mannyamorim Can you pass the stream key dynamically?

[mannyamorim]

@mannyamorim Can you pass the stream key dynamically?

@haerong22 I am assuming that you want to relay an RTMP stream to an RTMPS target with a dynamic stream key. This is possible using the notify module and the on_publish callback. You can refer to the existing documentation. In this PR I added support for you to use a URL that contains a domain name instead of just a static IP.

Essentially what happens here is that you can configure the module to execute a callback to a certain URL. If that URL returns a redirect then the returned URL with the stream key appended to it, is used as the target for the rtmp relay.

If you take a look at the sample configuration for the PR. The application test2 shows how you can build a relay system with multiple targets where all URLs and keys are dynamic. In this case the actual URLs with the keys are served by Nginx in the http block, however you can use anything you want to serve the URLs with the keys dynamically.

[haerong22]

@mannyamorim omg! so fast... thanks. but i can't understand ...

i want OBS -> NGINX -> YOUTUBE
how can i pass youtube stream key???
i try OBS -> NGINX : rtmp://localhost:1935/test/key?youtube=youtube-stream-key

i want like this

location /youtube {
            rewrite ^.*$ rtmps://a.rtmps.youtube.com/live2/$arg_youtube? permanent;
 }

rtmp {
    ...
    server {
         
         application test {
                ...
                push rtmp://127.0.0.1:1935/youtube;
                ...
         }

         application youtube {
            live on;

            on_publish http://127.0.0.1:8080/youtube;
        }
    }
}

[mannyamorim]

@haerong22 Sorry, that configuration is not directly supported by this module, however with a bit more work you could build something like this.

When you use the on_publish command, it appears that any arguments that you use should get passed through. Maybe if you try something like https://github.com/calio/form-input-nginx-module you can get the rewrite command to read in the arguments into a variable so you can do something like your example above.

This is just an idea, not something tested or supported by me, so you will need to experiment a bit.

[spjoes]

May need a different pull request but while you're at it with port 443 and rtmps, why not allow https on notify (like on_publish, on_play, on_publish_done, etc.)?
I've tried adding it myself before but keep getting "invalid port in url" and such. Not sure why.
Even tried one of the other pull requests on nginx-rtmp-module. Same issue.

[mannyamorim]

May need a different pull request but while you're at it with port 443 and rtmps, why not allow https on notify (like on_publish, on_play, on_publish_done, etc.)? I've tried adding it myself before but keep getting "invalid port in url" and such. Not sure why. Even tried one of the other pull requests on nginx-rtmp-module. Same issue.

@spjoes it is not trivial to add https support, rather it would require a fair amount of new code and testing. At this point I have no plans to add any such support to my fork. I understand that it would be more convenient, however you can simply have Nginx function as an RP for any https endpoints that you need to call out to, so this limitation shouldn't really impact the functionality of the module.

[spjoes]

@spjoes it is not trivial to add https support, rather it would require a fair amount of new code and testing. At this point I have no plans to add any such support to my fork. I understand that it would be more convenient, however you can simply have Nginx function as an RP for any https endpoints that you need to call out to, so this limitation shouldn't really impact the functionality of the module.

I was actually never able to get a RP working. Tried for a few hours and figured it'd be a better, cleaner, and (at the time) a quicker fix to just have https as a feature. I'll continue to try with proxy_pass and such but just native https support would be a great feature

[jazib93]

@mannyamorim Impressive work you've done. I've been building a streaming app. I have configured the rtmp module with nginx on Fedora 40. Streams are being pushed from https://blueirissoftware.com/ to my server at rtmp://ip-address/hls/dynamic-keys for testing which works fine.

Now I want to implement RTMPS and want to get the streams by giving rtmps://ip-address/hls/dynamic-keys to the third party softwares. But I'm not able to setup rtmps with nginx configuration. When I try to implement ssl within RTMP block. It gives [ssl parameter not found](nginx: [emerg] the invalid "ssl" parameter in).

rtmp {
server {
listen 1935 ssl;
chunk_size 4096;

           ssl_certificate /path/to/my/certificate/file/fullchain.pem
           ssl_certificate_key /path/to/my/certificate/file/privkey.pem
           
           Other configurations here;
   }

}

[mannyamorim]

@jazib93 the functionality that you're looking for is not present in this PR. I did eventually built it though and it's available from my fork: https://github.com/mannyamorim/nginx-rtmp-module however, I'm not planning on submitting it for inclusion here unless there is some progress made on reviewing/merging this first PR.

I should also give the disclaimer that as the code is not submitted for inclusion there is no formal documentation and I may or may not be able to support it in the future.

That being said the usage is straightforward. You can refer to this file: https://github.com/mannyamorim/nginx-rtmp-module/blob/master/ngx_rtmp_ssl_module.c for the list of supported directives. At a basic level the following could be used as a starting point for the config:

rtmp {
	server {
		listen 1935;
		listen 1936 ssl;

		ssl_certificate -----;
		ssl_certificate_key -----;
		ssl_dhparam -----;
		ssl_protocols TLSv1.2 TLSv1.3;
		ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
		ssl_verify_client on;
		ssl_verify_depth 1;
		ssl_client_certificate -----;

		# applications here
	}
}

[jazib93]

@mannyamorim Okay I have re-complied nginx with your Repo and did the above configuration but I'm not able to push my stream to my server on 'rtmps://' using ffmpeg and neither with OBS. I'm getting the error below

Stream #0:1: Audio: pcm_f32le, 44100 Hz, mono, flt, 1411 kb/s
[rtmps @ 0x600002bd4480] Cannot read RTMP handshake response
[out#0/flv @ 0x600002cc46c0] Error opening output rtmps://ip-address/hls/stream-key: End of file
Error opening output file rtmps://ip-address/hls/stream-key.
Error opening output files: End of file

I'm trying to understand how will the RTMP module will know that its accepting rtmps protocol as well. Your input will be highly appreciated.

Thanks

[mannyamorim]

@jazib93 This sounds like a configuration issue. Please share your Nginx configuration as well as the commands used to invoked ffmpeg or the config used for OBS so that I can try and see what may need to be changed.

[jazib93]

@mannyamorim here is my configuration and the FFMPEG command

rtmp {
        server {
                listen 1935;
                listen 1936 ssl;
                chunk_size 4096;
                
                ssl_certificate /etc/letsencrypt/live/domain-name/fullchain.pem;
                ssl_certificate_key /etc/letsencrypt/live/domain-name/privkey.pem;

                ssl_protocols TLSv1.2 TLSv1.3;
                ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
                ssl_verify_depth 1;
                
                application hls {
                        live on;
                        hls on;
                        hls_path /var/www/html/rtmp/hls/;
                        hls_nested on;
                        hls_type live;
                        hls_fragment_naming system;
                        hls_continuous on;
                        hls_playlist_length 60s;
                        hls_cleanup on;
                       
                        record all;
                        record_path /var/www/html/rtmp/hls-recordings;
                        record_unique on;
                        exec_record_done hls-recording.sh $path;
                        record_interval 120s;
               }
        }
}

This is the ffmpeg command that I ran from my mac:
ffmpeg -f avfoundation -framerate 30 -video_size 1280x720 -i "0:0" -vcodec libx264 -preset veryfast -maxrate 3000k -bufsize 6000k -pix_fmt yuv420p -g 50 -acodec aac -b:a 128k -ar 44100 -f flv rtmps://ip-address/hls/stream-key

When I use "rtmp://" with the same command, it works fine.

[mannyamorim]

@jazib93 You will need to set the port in your ffmpeg command to the one that is specified in Nginx:

ffmpeg -f avfoundation -framerate 30 -video_size 1280x720 -i "0:0" -vcodec libx264 -preset veryfast -maxrate 3000k -bufsize 6000k -pix_fmt yuv420p -g 50 -acodec aac -b:a 128k -ar 44100 -f flv rtmps://ip-address:1936/hls/stream-key

Alternatively you could change the Nginx config to listen on the default port 443 by doing listen 443 ssl;

[jazib93]

@mannyamorim Thank you so much, it worked by mentioning the port along with the IP address. I cannot use port 443 for nginx as its in use with apache for other apps.