Red5 now supports RTMPS, many of you will be very happy about this new feature. Previously, users had to use stunnel or some other "hack" to implement this feature that FMS supports. RTMPS was easy to implement after I realized RTMPS is RTMPT over SSL; this is a big revelation, since I started out thinking it was RTMP via SSL/TLS. So without further rambling, here is how to setup RTMPS in Red5 at zero cost. I would rate the procedure below as Intermediate level.
Create a self-signed certificate
There are many articles on the web which describe how to complete this first task, but I will explain how to do this with the tool included with Java. Execute this command from your console:
keytool -genkey -alias red5 -keyalg RSA -keysize 512 -validity 3650 \
-keystore keystore -keypass password -storepass password \
-dname "CN=localhost,OU=Red5,O=Red5,L=Henderson,ST=NV,C=US"
You may change a few of these options if you prefer, but I suggest that you only change some of the DName fields: Location (L), State (ST), and Country (C). Otherwise the feature may not work. Also note that if you are going to use RTMPS on the Internet, you will probably want to get a CA signed certificate.
Upon completion of this task you will have a file named "keystore", keep track of this since you will need it later on.
Get Red5
The RTMPS feature is not released as part of the current stable server (version 0.7.0) it is however available via SVN starting with revision 2819. This means you have to get an Subversion client, grab the source, build, and deploy; I will not be covering all these steps in this post. Go to http://osflash.org/red5 for links and information on Red5.
Configuration
Once you have Red5, copy the keystore file you created with step one into your conf directory (overwrite the current keystore file in the directory if prompted). The configuration file containing the parameter for RTMPS is in conf/red5-core.xml and you will need to update the RTMPS section if you changed the keystore password. The other configuration file to make note of is red5.properties, it contains the port assignments for the supported protocols. Make sure that you have your ports configured correctly; Adobe suggests that RTMPS be defaulted to port 443, normally this port is reserved for HTTPS. If you change this port to something other than 443 you will need to update your NetConnection urls with the correct port.
For Advanced users, there are a few other options you may specify on your RTMPS connection such as the supported ciphers. You can find additional information on this here. The connectioProperties section is where you would place these additional key / value pairs.
Engage
Start the server and test your applications. I personally tested with the oflaDemo that comes bundled in Red5. Go to http://localhost:5080/demos/oflaDemo.swf and change the rtmp url to rtmps://localhost/oflaDemo et voila! This of course assumes that you didnt change any ports.

Debugging
To see what is going on with SSL at a really low level you can enable this option in your startup:
-Djavax.net.debug=ssl
It will cause additional information to be displayed on the console.