Merge pull request #14 from drowe67/dr-aug-2024

August 2024 Test Campaign #2

README.md

@@ -4,7 +4,7 @@ A hybrid Machine Learning/DSP system for sending speech over HF radio channels.
 
 ## Scope 
 
-This repo is intended to support the authors experimental work, with just enough information for the advanced experimenter to reproduce aspects of the work.  It is not intended to be a polished distribution for general use.  Unless otherwise stated, the code is this repo is intended to run only on Ubuntu Linux.
+This repo is intended to support the authors experimental work, with just enough information for the advanced experimenter to reproduce aspects of the work.  It is not intended to be a polished distribution for general use.  Unless otherwise stated, the code is this repo is intended to run only on Ubuntu Linux 22.
 
 # Quickstart
 
@@ -39,18 +39,18 @@ The RDOVAE derived Python source code is released under the two-clause BSD licen
 | multipath_samples.m | Octave script to generate multipath magnitude sample over a time/freq grid |
 | plot_specgram.m | Plots sepctrogram of radae modem signals |
 | radae_plots.m | Helper Octave script to generate various plots |
-| radio_ae.[tex|pdf] | Latex documenation |
+| radio_ae.[tex,pdf] | Latex documenation |
 | ota_test.sh | Script to automate Over The Air (OTA) testing |
 | Radio Autoencoder Waveform Design.ods | Working for OFDM waveform, including pilot and cyclic prefix overheads |
 | compare_models.sh | Builds loss versus Eq/No curves for models to objectively compare |
 | est_snr.py | Prototype pilot sequence based SNR estimator - doesn't work for multipath |
 | test folder | Helper scripts for ctests |
 | loss.py | Tool to calculate mean loss between two feature files, a useful objective measure |
 | ml_pilot.py | Training low PAPR pilot sequence |
-| stateful_decoder.[py|sh] | Inference test that compares stateful to vanilla decoder |
-| stateful_encoder.[py|sh] | Inference test that compares stateful to vanilla encoder |
-| radae_tx.[py|sh] | streaming RADAE encoder and helper script |
-| radae_rx.[py|sh] | streaming RADAE decoder and helper script |
+| stateful_decoder.[py,sh] | Inference test that compares stateful to vanilla decoder |
+| stateful_encoder.[py,sh] | Inference test that compares stateful to vanilla encoder |
+| radae_tx.[py,sh] | streaming RADAE encoder and helper script |
+| radae_rx.[py,sh] | streaming RADAE decoder and helper script |
 | resource_est.py | WIP estimate CPU/memory resources |
 
 # Installation
@@ -65,28 +65,34 @@ Supplies some utilities used for `ota_test.sh` and `evaluate.sh`
 ```
 cd ~
 git clone [email protected]:drowe67/codec2-dev.git
-cd codec2
+cd codec2-dev
 mkdir build_linux
 cd build_linux
 cmake -DUNITTEST=1 ..
 make ch mksine tlininterp
 ```
-(optional if using HackRF) manually compile codec2-dev/misc/tsrc.c
 
-# Building and Automated Tests
+## RADAE
 
-The `cmake/ctest` framework is being used as a build and test framework. The command lines in `CmakeLists.txt` are a good source of examples, if you are interested in running the code in this repo.
-
-To configure and run the cests:
+Builds the FARGAN vocoder and ctest framework, most of RADAE is in Python.
 ```
 cd radae
 mkdir build
 cd build
 cmake ..
 make
+```
+
+# Automated Tests
+
+The `cmake/ctest` framework is being used as a build and test framework. The command lines in `CmakeLists.txt` are a good source of examples, if you are interested in running the code in this repo. The ctests are a work in progress and may not pass on all systems (see Scope above).
+
+To run the cests:
+```
+cd radae/build
 ctest
 ```
-To list tests `ctest -N`, to run just one test `ctest -R inference_model5`, to run in verbose mode `ctest -V -R inference_model5`.  You can change the paths to `codec2-dev` and `opus` on the `cmake` command line:
+To list tests `ctest -N`, to run just one test `ctest -R inference_model5`, to run in verbose mode `ctest -V -R inference_model5`.  You can change the paths to `codec2-dev` on the `cmake` command line:
 ```
 cmake -DCODEC2_DEV=~/tmp/codec2-dev ..
 ```
@@ -245,9 +251,9 @@ BER tests are useful to calibrate the system, and measure loss from classical DS
 
 1. Testing OTA over HF channels. Using my IC7200 as the Tx station:
    ```
-   ./ota_test.sh wav/david.wav -g 9 -t -d -f 14236
+   ./ota_test.sh wav/david_vk5dgr.wav -g 6 -t -d -f 14236
    ```
-   The `-g 9` sample gives the `david.wav` sample a little more compression, this was ajusted by experiment, listening to the `tx.wav` file, and looking for signs of a compressed waveform on Audacity.  To receive the signal I tune into a convenient KiwiSDR, and manually start recording when my radio starts transmitting.  I stop recording when I hear the transmission end.  This will result in a wave file being downloaded.  It's a good idea to trim any excess off the start and end of the rx wave file. It can be decoded with:
+   The `-g 6` is the SSB compressor gain (default 6 so in this case optional); this can be adjusted by experiment, e.g. listening to the `tx.wav` file, and looking for signs of a compressed waveform on Audacity.  To receive the signal I tune into a convenient KiwiSDR, and manually start recording when my radio starts transmitting.  I stop recording when I hear the transmission end.  This will result in a wave file being downloaded.  It's a good idea to trim any excess off the start and end of the rx wave file. It can be decoded with:
    ```
    ./ota_test.sh -d -r ~/Downloads/kiwisdr_usb.wav
    ```
@@ -407,3 +413,86 @@ Using model 17 waveform:
 | Auxilary text channel | No | |
 | SNR measurement | No | |
 | Tx and Rx sample clock offset | 200ppm | e.g. Tx sample clock 8000 Hz, Rx sample clock 8001 Hz |
+
+# Web based Stored File Processing
+
+This section contains some notes on setting up a web server to run `ota_test.sh`.  The idea is to make it easier for non-Linux users to contribute to the stored file test program.  The general idea is a CGI script interfaces to `ota_test.sh` to perform the Tx and Rx processing.  We configure the web server so that the HTML forms and CGI scripts run in `~/public_html`.  The notes below are for Apache on Ubuntu 22. 
+
+1. The Python packages need to be available system wide , so `www-data` can use them: 
+   ```
+   sudo pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
+   sudo -u www-data python3 -c "import torch"
+   ```
+
+   ```
+   sudo pip3 install matplotlib
+   sudo -u www-data python3 -c "import matplotlib"
+   ```
+   The presence of the packages can be checked by mimicing the www-data user (the last line in each step above should return nothing if all is well).
+
+1. Configure Apache for CGI and serving pages from our `~/public_html` dir.
+   ```
+   sudo a2enmod cgid
+   sudo a2enmod userdir
+   sudo systemctl restart apache2
+   ```
+   We want html and cgi to run out of ~/public_html, so permissions have to be `755` and `www-data` has to be added to the users group.
+   ```
+   mkdir ~/public_html
+   chmod 755 public_html
+   sudo usermod -a -G <username> www-data
+   ```
+   To let CGI scripts run from ~/public_html I placed this in my `/etc/apache2.conf`:
+   ```
+   <Directory "/home/david/public_html">
+      Options +ExecCGI
+      AddHandler cgi-script .cgi
+   </Directory>  
+   ```   
+   Then restart apache as above.
+
+1. Create sym links to HTML/CGI scripts in `radae` repo, this allows the script to be part of the RADAE repo:
+   ```
+   cd ~/public_html
+   ln -s ~/radae/public_html/tx_form.html tx_form.html
+   ln -s ~/radae/public_html/tx_process.cgi tx_process.cgi
+   ``` 
+
+1. Note that files created when the CGI process run (e.g. `/tmp/input.wav`) get put in a sandbox rather than directly in `/tmp`.  This is a systemd security feature.  You can find the files with:
+   ```
+   sudo find /tmp -name input.wav | xargs sudo ls -ld
+   -rw-r--r-- 1 www-data www-data 3918458 Aug 15 15:28 /tmp/systemd-private-2fcf85ad243b4da08d79d2e27e0375af-apache2.service-vDE2Dg/tmp/input.wav
+   ```
+
+1. Apache error log, good for viewing `ota_test.sh` progress and spotting any issues:
+   ```
+   tail -f /var/log/apache2/error.log
+   ```
+
+# Real Time PTT
+
+WIP notes
+
+## Real Time decode using KiwiSDR
+
+1. Install pulse audio null module
+   ```
+   pactl load-module module-null-sink sink_name=vsink
+   ```
+1. Start your web browser, and open a tab to a KiwiSDR.
+1. Open `pavucontrol`, *Playback* tab, send web browser sound to NULL module.  Audio from web browser should go silent.
+1. We take the audio from the null device monitor output for the input to the RADAE Rx:
+   ```
+   parec --device=vsink.monitor --rate=8000 --channels=1 | python3 int16tof32.py --zeropad | python3 radae_rx.py model19_check3/checkpoints/checkpoint_epoch_100.pth -v 2 --auxdata | ./build/src/lpcnet_demo -fargan-synthesis - - | aplay -f S16_LE -r 16000
+   ```
+1. Try transmitting a RADAE signal:
+   ```
+   ./ota_test.sh -t radae_test.raw -d -f 7175
+   ```
+   Where `radae_test.raw` is a RADAE-only sample (i.e. without the chirp and SSB, copied from a temp file generated by `ota_test.sh -x`). If you can't open the SSB radio playback device to radio try closing `pavucontrol`.
+1. Other useful pulse audio commands:
+   ```
+   pactl list sinks short
+   pactl list sources short
+   pactl list modules
+   ```

doc/radae.tex

@@ -1165,6 +1165,14 @@ \subsection{Sync State Machine}
 \label{tab:acq_req}
 \end{table}
 
+TODO:
+\begin{enumerate}
+\item discuss EOO system.
+\item EOO system, not always reliable, higher threshold to avoid false unsync at low SNRs.  Art low SNRs its acceptable to use UW errors to unsync. Add to table above?
+\item State diagram with EOO, run on counter, UW errors. 
+\item Is run on counter required with UW error system?
+\end{enumerate}
+
 \section{Auxiliary Data}
 
 In existing FreeDV modes, a few bits/frame are allocated for supplementary information such as a unique word (UW) to support reliable sync, and low bit rate (e.g. 25 bits/s) data.  In existing FreeDV modes we use digital modulation, so it is easy to allocate some of these bits to low bit rate data.  For ML based waveforms, the network controls the PSK symbols being sent across the channel. One approach is to add additional OFDM carriers dedicated to data, however this may upset the PAPR optimisation, and increases the bandwidth of the signal.  A single carrier would have poor resilience to multipath.
@@ -1192,7 +1200,7 @@ \section{Auxiliary Data}
 \end{table}
 
 \begin{figure}[H]
-\caption{Loss versus $E_q/N_0$ for various models. \emph{mode17\_check3} is coincident with  \emph{mode17}, indicating very little penalty for the injection of the auxiliary data over a range of SNRs.}
+\caption{Loss versus $E_q/N_0$ for various models. \emph{mode17\_check3} is coincident with  \emph{model17}, indicating very little penalty for the injection of the auxiliary data over a range of SNRs.}
 \label{fig:loss_eqno_models}
 \begin{center}
 \input loss_EqNo_models.tex

doc/stored_file_test.md

@@ -0,0 +1,94 @@
+# Stored File Test Procedure
+
+This document is a test procedure for the August 2024 RADAE stored file test campaign.  The general idea is to take a 10 second sample of input speech, then send it over a HF radio channel as compressed SSB and RADAE.  This allows a side by side comparison using the same speech, over approximately the same channel conditions.  Using a stored file makes it possible to repeat the experiment in a controlled fashion over several trials, for example with varying power levels of different receiver locations. As background a similar campaign was conducted in [April 2024](https://freedv.org/?p=595).
+
+If you are comfortable with Ubuntu Linux you can install the RADAE tools required on your machine and perform the processing yourself. If you are uncomfortable with Linux the RADAE processing can be performed using a web site.  See Web Based Processing section below as an alternative to using `ota_test.sh`.
+
+## Preparing a file for Tx
+
+1. If you wish to perform the RADAE processing on your Ubuntu Linux machine, [Install](../README.md#installation) the RADAE software.
+1. Record a wave file of your own voice, for example, "Hello, this is VK5XYZ testing the radio Autoencoder 1 2 3 4".
+1. The wave file format required is 1 channel, 16 kHz, 16 bit signed integer.
+1. We suggest about 10 sec long but feel free to experiment.  The length is not critical.
+1. Try to make the peak level between about half way and the clipping level.
+   ![Peak level example](stored_file_level.png)
+1. Use a headset microphone and try to avoid room echo and background noise. Don't use a laptop microphone.
+1. As examples, there are samples from other hams in `radae/wav`
+1. Use `ota_test.sh` to create a `tx.wav`, this consist of chirp-compressed SSB-radae:
+   ```
+   ota_test.sh -x vk5xyz.wav
+   ```
+1. You can listen to and plot `tx.wav` with your favorite waveform editor, you can see the signals are adjusted to have the same peak level. The SSB compression gain can be adjusted using the `-g` option; `-g 6` is the default.  Trying going up or down 3dB.  A quieter sample may benefit from more compression.
+   ![Peak level example](stored_file_tx.png)
+
+## Transmitting your sample
+
+1. Configure your SSB radio with voice compressor off.  The Tx audio path must be "clean" with no additional processing.
+1. The sound card levels should be adjusted to "just move the ALC".
+1. Tune the remote receiver (I use a KiwiSDR) to your SSB radio frequency.  You need to be within +/- 50 Hz for the RADAE receiver to acquire.
+1. Start transmitting. You can do this manually by playing tx.wav through your transmitter, or use `ota_test.sh`
+   ```
+   ./ota_test.sh wav/david_vk5dgr.wav -d -f 14236
+   ```
+   You can adjust the hamlib rig and serial port with command line options, to get help: `ota_test.sh -h`
+1. After you start transmitting quickly start the KiwiSDR recording.
+1. When you hear transmission stop on the KiwiSDR, stop recording.
+1. The KiwiSDR file will be downloaded.
+1. Place a serial number in front of the downloaded file to easily identify it e.g. `vk5xyz_14_`. Make your own notes of the conditions for that sample (e.g. rx station location, distance, power level, anything else you think is relevant)
+1. It's useful to load the file into your waveform viewer.  I find spectrogram mode useful.
+   ![Peak level example](stored_file_rx.png)
+1. The RADAE receiver will search for the location of the chirp in the first 10 seconds of the sample.  Make sure there is no more than 6 seconds of noise before the chirp starts.  If necessary, edit the file by removing any excess before the chirp starts.
+1. If there is more than a few seconds after the RADAE signal stops, clip that off the sample too.
+1. Process the received sample:
+   ```
+   ./ota_test.sh -r ~/Downloads/14_sdr.ironstonerange.com_2024-08-08T05_10_14Z_7175.00_lsb.wav
+   ```
+1. This will locate the chirp, print the C/No and SNR in dB, and generate several other files in the same directory, `_ssb.wav`, `_radae.wav` and a spectrogram `_spec.jpg`.  The mesured C/No and SNR will be in `_report.txt`.
+1. Note the C/No, SNR, and listen to the results, comparing SSB to RADAE.
+1. Try to collect some interesting results, for example:
+   * Different channels (ground wave, NVIS, DX) and power levels
+   * Any cases where RADAE fails to acquire (no decode)
+   * Intercontinental DX, fast and slow fading
+   * co-channel interference (SSB on top of or near RADAE)
+   * interference from carriers
+   * try listening through small laptop speaker, large external speakers, headphones
+   * try different microphones
+   * try different langauges
+   * different radios and rig interfaces
+   * Old VFO radio with some drift
+
+## Web Based Processing
+
+A web site has been developed to perform the `ota_test.sh` processing.  The URL will be made available to the test team.
+
+1. In the Tx Processing section Browse to the file you want to encode for Tx, then press Process.  In a few seconds hopefully `tx.wav`` will be downloaded.  Use your own sample (prepared as described above), or for testing choose a sample form the RADAE Git repo [wav](https://github.com/drowe67/radae/tree/main/wav) folder.
+2. To test the Rx Processing Browse to the `tx.wav`` you just generated. Press Process, in about 10 seconds a zip file will be returned to you with the SSB and RADAE, a spectogram, and a report file.
+3. If you get this far, Transmit `tx.wav`` over your SSB radio to a remote Rx (as described above), and try processing the received file.
+
+## Receiver log fields
+
+For example:
+
+```
+43 state: candidate  valid: 1 0  2 Dthresh:     9.55 Dtmax12:    15.67     4.01 tmax:  596 tmax_candidate:  596 fmax: -12.50
+44 state: candidate  valid: 1 0  3 Dthresh:     9.34 Dtmax12:    15.08     4.01 tmax:  596 tmax_candidate:  596 fmax: -12.50
+45 state: sync       valid: 1 0 25 Dthresh:     8.45 Dtmax12:    12.70     2.56 tmax:  596 tmax_candidate:  596 fmax: -10.98 auxbits: [0 0 0] uw_errors: 0
+46 state: sync       valid: 1 0 25 Dthresh:     8.35 Dtmax12:     8.92     1.48 tmax:  596 tmax_candidate:  596 fmax: -10.98 auxbits: [0 1 0] uw_errors: 1
+47 state: sync       valid: 0 0 25 Dthresh:     8.17 Dtmax12:     5.17     1.45 tmax:  596 tmax_candidate:  596 fmax: -10.89 auxbits: [0 0 0] uw_errors: 1
+48 state: sync       valid: 0 0 24 Dthresh:     7.97 Dtmax12:     3.52     1.47 tmax:  595 tmax_candidate:  596 fmax: -10.83 auxbits: [0 0 1] uw_errors: 2
+49 state: sync       valid: 0 0 23 Dthresh:     7.80 Dtmax12:     4.95     1.60 tmax:  596 tmax_candidate:  596 fmax: -10.74 auxbits: [1 0 0] uw_errors: 3
+50 state: sync       valid: 0 0 22 Dthresh:     7.64 Dtmax12:     7.06     2.37 tmax:  595 tmax_candidate:  596 fmax: -10.72 auxbits: [0 0 0] uw_errors: 3
+51 state: sync       valid: 1 0 21 Dthresh:     7.52 Dtmax12:     8.37     1.06 tmax:  596 tmax_candidate:  596 fmax: -10.81 auxbits: [1 0 0] uw_errors: 4
+```
+
+| Field | Description |
+| ---- | ---- |
+| state | Sync state machine state |
+| valid | Input to state machine: valid pilot seq, valid end of over sequence, run on counter |
+| Dthresh | current threshold for detection of pilot sequence |
+| Dtmax12 | current maxima of pilot sequence, end of over sequence |
+| tmax | current timing estimate |
+| fmax | current frequency offset estimate |
+| auxbits | Auxillary bits received, will be all 0s if no bit errors |
+| uw_errors | current count of unique word (auxillary bit) errors, reset every second |
+