BNCs are objectively much better than RCA connectors at the Radio Frequencies (RF) that digital signals run at. As another aside I find the RCA connector wholly unsatisfying for analog audio also. RCAs were a poor audio connector design and an even worse choice for RF connections. BNCs were designed for high quality, high frequency instrumentation and are vastly better suited to digital signals. You can pretty easily replace any RCA connectors used for digital inputs or outputs with BNCs. Be sure to use true 75 Ohm BNCs with teflon insulators and keep the leads as short as possible.
Given a choice, avoid carrying S/PDIF over Toslink. Most Toslink transceivers have much more inherent jitter than the wired Coaxial interfaces on BNC or RCA. I would not recommend using a player with only Toslink outputs as a transport. If you have a service manual or a good eye for circuits you may be able to tap off the S/PDIF before it gets to the Toslink transceiver and wire up a bulkhead mount BNC connector for S/PDIF output instead. Keep the wire short and as close as possible to the board and tap off after coupling capacitors or transformers but before output opamps.
For a comparison of interface sound quality, my Pioneer DV-414 has both coaxial and Toslink digital outputs. Sending both to an Assemblage D2D-1 outboard jitter reducer/upsampler/interpolater and DAC-3, the Toslink still had a lot worse sound quality. It was fuzzy, hashy, grainy, etc. This is with the dual PLL jitter reduction, low-jitter I2Se interface, sample rate upconversion, etc. applied to both signals. The coax is much better: clearer, smoother, more continuous and solid. Again this is even after the pretty extensive processing to clean up the signal that this gear provides. So the Toslink was doing a lot of damage to the same signal sent simultaneously from the same transport, and even a lot of digital signal cleanup could not undo the damage. The obvious conclusions are that the DAC is important, the transport also important, but the interface type is very important. A high jitter interface like Toslink can definitely harm the signal, and should be avoided if you want to get the best sound.
One advantage of Toslink or other optical fibers that one might suppose is the isolation of electrical grounds, which would prevent ground loops between equipment. Ground loops can be a carrier for noise. However since most good audio equipment uses isolation transformers on digital inputs and outputs, optical fibers are often not an advantage in preventing ground loops.
Please see also my comments on Digital Interface Formats.