This all looks thoroughly broken. What do you see if you don't feed any signal at all? So pure silence, no stereo pilot. I'm interested in that last graph.
To test modulation levels, try the sine wave generator in Stereo Tool and try some frequencies from 30 Hz up to 60 kHz, and check if the measured levels are the same. They should be. If there are any weird filters still active, you should see them if you do this. It's hard to determine what's going on with the 19 kHz tone because if the transmitter is doing something with stereo or RDS, one would expect spikes at 38 and 57 kHz - so I'm curious what happens with those spikes if you feed a somewhat different frequency, for example 20 kHz.
If they are still there you're seeing some form of harmonics distortion, and that's going to be extremely hard to get rid of. If you don't, there might still be something running inside the transmitter that attempts to create a stereo pilot.
Oops! My bad for the late reply. I thought this thread has died.
This is what it looks like with no stereo pilot:
I don't know what that little line is on the left... Then again, I did mess with the reference frequency so the little line could be in the wrong location.
The signal is not exactly in the frequency of broadcast on the RTLSDR. So I'd assume I change the reference frequency to its at the frequency of broadcast.
I tried comparing the RF FTT FM boadcast of the transmitter to an commercial FM station with no audio, only stereo pilot and RDS. It looks a little different..
Either way: I don't think the transmitter is creating its own stereo pilot. I would assume the transmitter just has repeated frequencies... I am clueless about what is "normal" for this FM transmitter oh my god. I wish I was an FM Transmitter expert.
Well, there definitely is a filter because if I try a sine wave at 30khz up to 60khz, I end up with different levels of modulation. Half of it is lower, and half of it is higher....
I mean seriously, if I use a square wave at 30khz its a mess:
Though, due to overshoots... I mean, it goes off the frequency bandwidth....
EDIT: It's annoying to think about but basically, I got scammed by PCS electronics. They advertise a good signal off of their FM transmitter but I get crap.