Asset UPnP for Raspberry pi

[dunc]

Re: Asset UPnP for Raspberry pi

...

So the fstab entry is currently

//192.168.1.3/media /home/pi/nas cifs rsize=3843360,guest,_netdev,sec=ntlm 0 0

So the read buffer is at the max of 3.6MBytes as taken from the $ sudo sysctl -a |grep mem command, which returns with min 4096, default 87380, max 3843360 for the net.ipv4.tcp_rmem entry. However the net.core.rmem_default and net.core.rmem_max entries are both 163840 (160KBytes), so is the max that the read buffer can be set to?

...

Hi Simon (simes_pep)

My understanding from the Linux docs is that net.core.rmem_default and net.core.rmem_max entries both take effect only for protocols that do not have more specific tunables available. For example, in the rmem case TCP has more specific tunable with the net.ipv4.tcp_rmem setting. For UDP, which doesn't have a more specific tunable, the net.core.rmem_* settings are relevant.

For the CIFS rsize setting you have set, this should be taking effect as this is the same as the max configued in net.ipv4.tcp_rmem. You will of course have to remount the CIFS share for the change to take effect (I'm assuming you have done this).

...

Anyway when transcoding a 24/192 file the upstream I/O is a constant 1.1-1.2 MB/s (9,216 kb/s for the 24/192 file is 1.125 MB/s), so the correct and necessary I/O rate for the uncompressed data stream.
The downstream I/O, i.e. into the RPi from the NAS is peaking at 3.8MB/s during a read, and then given that it is has 3.6MB of data to read from the buffer to serve the outgoing stream, falls back to a minimal 30KB/s level.
So there is no read buffer exhaustion in the RPi. CPU processing is around 30% to conduct the transcoding.

...

Interesting. When transcoding a 192/24 FLAC to WAV on my Pi I see a max CPU of about 18% for the Asset process and about 2.3% for the smbd mount process, and the Pi has about 78% idle CPU. I'm running the cutdown Minibian image instead of standard Raspbian image, however I still wouldn't have expected such a big difference in CPU usage. Could be a difference in the compression setting in your FLACs vs mine perhaps?

...

Do we need to look at the net.ipv4.tcp_wmem settings, which are 4096, 16384, 3843360, so defaulting to 16k
The net.core.wmem_default & net.core.wmem_max are also at 163840 (160K)

...

I played around with default size in net.ipv4.tcp_wmem, using 4k, 8k, 16k, 1MB and 2MB, and I saw the behaviour change most significantly with 1MB and 2MB sizes - the Naim's buffer % initially improving, then degrading to much worse than with the smaller settings, so I abandoned this line of tuning.

Cheers,
Dunc

[dunc]

Re: Asset UPnP for Raspberry pi

Hi Simon (simes_pep)

My understanding from the Linux docs is that net.core.rmem_default and net.core.rmem_max entries both take effect only for protocols that do not have more specific tunables available. For example, in the rmem case TCP has more specific tunable with the net.ipv4.tcp_rmem setting. For UDP, which doesn't have a more specific tunable, the net.core.rmem_* settings are relevant.

For the CIFS rsize setting you have set, this should be taking effect as this is the same as the max configued in net.ipv4.tcp_rmem. You will of course have to remount the CIFS share for the change to take effect (I'm assuming you have done this).



Interesting. When transcoding a 192/24 FLAC to WAV on my Pi I see a max CPU of about 18% for the Asset process and about 2.3% for the smbd mount process, and the Pi has about 78% idle CPU. I'm running the cutdown Minibian image instead of standard Raspbian image, however I still wouldn't have expected such a big difference in CPU usage. Could be a difference in the compression setting in your FLACs vs mine perhaps?




I played around with default size in net.ipv4.tcp_wmem, using 4k, 8k, 16k, 1MB and 2MB, and I saw the behaviour change most significantly with 1MB and 2MB sizes - the Naim's buffer % initially improving, then degrading to much worse than with the smaller settings, so I abandoned this line of tuning.

Cheers,
Dunc

simes_pep - sorry I put you wrong in this post - some docs on the web indicate that net.ipv4.tcp_rmem overrides net.core.rmem_max and net.core.rmem_default, and some say they do not. I finally found another doc that says its actually both - the default value in net.ipv4.tcp_rmem overrides net.core.rmem_default for TCP sockets, however the max value in net.ipv4.tcp_rmem DOES NOT override net.core.rmem_max. The same applies to wmem it seems.

In the case of the CIFS mount rsize buffer tuning I did a few days ago there was a difference between rsize of 384k and 512k, and I'm sure I left net.core.rmem at 160k. This puzzled me at first, however from what I've read since, it appears the the rsize and wsize mount options for CIFS/SAMBA shares are for the cifsd buffer, and is therefore not related to or restricted by the socket window/buffer sizes.

In the case of the tcp_wmem tuning I did however, this is interesting, as the range of defaults I tested did exceed the wmem_max of 160k, so I'll have to retry this testing sometime.

Cheers,
Dunc

[simes_pep]

Re: Asset UPnP for Raspberry pi

Hi @Dunc, @Batleys,
Many thanks for the input/observations.

From what I have observed the CIFS mount to the Gigabit enabled NAS is capable of serving the data processing required by the RPi for the transcoding without buffer exhaustion. It is the RPi to ND5 connection that is not able to get the packets into the Naim processing buffer fast enough, and is being exhausted. This is separate to the TCP buffers in the NIC interfaces (tcp_wmem, tcp_rmem) in both the RPi and ND5.
So I need to look at/improve this connection. I am investigating in getting a second switch (a small 5 port switch), so I can move the RPi 'closer' to the ND5, with the RPi->NAS connection over the Mains Plugs and the RPi->ND5 over normal twisted pair, instead of the other way round.

However in the meantime, all FLAC versions are playing perfectly and sounding great.
Simon

[simes_pep]

Re: Asset UPnP for Raspberry pi

Following the very useful input from @dunc and @Batleys, I have updated the Configuration Guide to include a section on CIFS buffers

Latest version in my Dropbox @ https://www.dropbox.com/sh/bu8etep76nu4v5z/aMw624ZA5U

Thanks,
Simon.

[dunc]

Re: Asset UPnP for Raspberry pi

Hi @Dunc, @Batleys,
Many thanks for the input/observations.

From what I have observed the CIFS mount to the Gigabit enabled NAS is capable of serving the data processing required by the RPi for the transcoding without buffer exhaustion. It is the RPi to ND5 connection that is not able to get the packets into the Naim processing buffer fast enough, and is being exhausted. This is separate to the TCP buffers in the NIC interfaces (tcp_wmem, tcp_rmem) in both the RPi and ND5.
So I need to look at/improve this connection. I am investigating in getting a second switch (a small 5 port switch), so I can move the RPi 'closer' to the ND5, with the RPi->NAS connection over the Mains Plugs and the RPi->ND5 over normal twisted pair, instead of the other way round.

However in the meantime, all FLAC versions are playing perfectly and sounding great.
Simon

Hi Simon, no problem at all.

That matches with my observations as well. That is a good idea to move your Pi to where the Naim is, relative to the LAN over mains power devices. If you still run into issues streaming from the NAS with this setup you may be able to adjust the rsize buffer on the CIFS mount to suit, as I did. Since tuning my CIFS mount, I've been happily using my Pi with Asset on the same switch as my NDX for streaming WAV source files as well as FLAC and transcoding on the Pi. Overall this is a great combo.

For what its worth - the problem I was seeing was, I think, down my small office home office network switch getting congestion that was beyond its buffering capabilities when, and only when, I was streaming a WAV source file via Asset on the Pi (there were no problems streaming FLAC source to the Pi and transcoding to WAV with this setup). I came to this conclusion after the other Simon (Batleys) mentioned earlier that he has no problems with this scenario. When comparing our setups I see the key difference is Simon is using an enterprise grade switch. Comparing the most recent specs I found for my device (NetGear GS108) and the Cisco switch Simon used (Cisco 2960 from memory), the Cisco had a much larger buffer per port (192k on the GS108 and 512k or greater on the Cisco), slightly lower latency, and the total switching capacity was higher (though this last spec is down to 16 ports vs 8 ports). I think if latency was the issue in my case, then moving the Pi to another switch on my network would have made that worse, instead it improved things. Therefore I think it was more likely the buffer sizes - ie if I upgraded my switch to a more capable model with larger buffer per port, that would also solve my scenario, ie it would raise the threshold at which congestion could occur on any given port. I still think the Naim is misbehaving at the TCP layer as this issue only occurs when the Naim is on the same switch, ie if the Pi is on another switch, it is still only the link to the Naim that has a constantly high Send-Q. I suspect this is possibly a lack of window scaling support on the Naim such that it can't adjust to the network conditions, though I won't be able to confirm that until I get some time to monitor and analyse the traffic on the Pi during this scenario.

Good luck with getting your system working with your planned setup!

Cheers,
Dunc

[Batleys]

Re: Asset UPnP for Raspberry pi

Simon
Very, very good - I'd be interested in reading the feedback of a non techie - it all appears clear to me - but then I guess I am not a great reference.
BTW being (very) picky - on page 4 the broadcast address is an IP address rather than an IP range and the subnet mask is simply a mask (of bits) defining the size of the network address and not itself an address, and finally the term 'gateway' is the original and now alternate term to a 'router' and for the most part they can be used interchangeably - they mean the same thing.
Simon

[simes_pep]

Re: Asset UPnP for Raspberry pi

No problem - thank you for the feedback and duly updated. My Unix/Networking is a little rusty, which is why I wanted to capture all the steps involved in a single document, incase I forget it all again.

Simon.

[simes_pep]

Re: Asset UPnP for Raspberry pi

Hi Simon, no problem at all.

That matches with my observations as well. That is a good idea to move your Pi to where the Naim is, relative to the LAN over mains power devices. If you still run into issues streaming from the NAS with this setup you may be able to adjust the rsize buffer on the CIFS mount to suit, as I did. Since tuning my CIFS mount, I've been happily using my Pi with Asset on the same switch as my NDX for streaming WAV source files as well as FLAC and transcoding on the Pi. Overall this is a great combo.

For what its worth - the problem I was seeing was, I think, down my small office home office network switch getting congestion that was beyond its buffering capabilities when, and only when, I was streaming a WAV source file via Asset on the Pi (there were no problems streaming FLAC source to the Pi and transcoding to WAV with this setup). I came to this conclusion after the other Simon (Batleys) mentioned earlier that he has no problems with this scenario. When comparing our setups I see the key difference is Simon is using an enterprise grade switch. Comparing the most recent specs I found for my device (NetGear GS108) and the Cisco switch Simon used (Cisco 2960 from memory), the Cisco had a much larger buffer per port (192k on the GS108 and 512k or greater on the Cisco), slightly lower latency, and the total switching capacity was higher (though this last spec is down to 16 ports vs 8 ports). I think if latency was the issue in my case, then moving the Pi to another switch on my network would have made that worse, instead it improved things. Therefore I think it was more likely the buffer sizes - ie if I upgraded my switch to a more capable model with larger buffer per port, that would also solve my scenario, ie it would raise the threshold at which congestion could occur on any given port. I still think the Naim is misbehaving at the TCP layer as this issue only occurs when the Naim is on the same switch, ie if the Pi is on another switch, it is still only the link to the Naim that has a constantly high Send-Q. I suspect this is possibly a lack of window scaling support on the Naim such that it can't adjust to the network conditions, though I won't be able to confirm that until I get some time to monitor and analyse the traffic on the Pi during this scenario.

Good luck with getting your system working with your planned setup!

Cheers,
Dunc

Yes, I was doing some further reading on the CIFS 2.0 module, which is included in the latest Kernel builds, following a apt-get update/apt-get upgrade.

From the $ man mount.cifs pages for the CIFS 2.0 module, this confirms that the default buffer size is 16K. However if the NAS supports large POSIX reads, then the default is 60K, and the maximum is 16M. I have only tried, so far, with a the CIFS buffer up to 3843360 (3.6M), as I thought this was the maximum. Given that I am dedicating the RPi as the UPnP server, and has given the majority of the on-board memory to the CPU, I would prefer to have it use the maximum buffer possible.

I too suspect the NIC in the Naim Network Players has been set & tested for operation in a dedicated sub-network with a UnitiServer on the same switch. Which sort of explains some of the behaviour if the UPnP server is on a different or 'distance' sub-network. Hopefully my 5-port switch will arriving soon and I can adjust my setup, to reflect this configuration, and give my feedback.

[simes_pep]

Re: Asset UPnP for Raspberry pi

Was just wondering whether PerfectTunes could be ported to the Raspberry Pi, to run as 'server side' function, given it can take a long time to 'listen' to a complete music collection.

Thanks,
Simon.

[Spoon]

Re: Asset UPnP for Raspberry pi

It would not, as it is mainly a user interface.

[PeterP]

Re: Asset UPnP for Raspberry pi

If anyone is wondering why I don't run Asset for Linux on my NAS - I tried this and it also solves the 192/96 WAV streaming problem, however my NAS is using the unRAID NAS O/S which boots from a USB key into a RAM disk, so the Asset config doesn't survive reboots (Spoon - if you're reading this is there a way to tell Asset for Linux in the command line script to look elsewhere for its config file location?), hence why I'm using Asset for Pi.

Late reply-
You can start Asset by:
export home=<any path you want> && ./AssetUPnP
Overriding the home directory variable will cause Asset to store its configuration elsewhere.

[simes_pep]

Re: Asset UPnP for Raspberry pi

Yes, I was doing some further reading on the CIFS 2.0 module, which is included in the latest Kernel builds, following a apt-get update/apt-get upgrade.

From the $ man mount.cifs pages for the CIFS 2.0 module, this confirms that the default buffer size is 16K. However if the NAS supports large POSIX reads, then the default is 60K, and the maximum is 16M. I have only tried, so far, with a the CIFS buffer up to 3843360 (3.6M), as I thought this was the maximum. Given that I am dedicating the RPi as the UPnP server, and has given the majority of the on-board memory to the CPU, I would prefer to have it use the maximum buffer possible.

I too suspect the NIC in the Naim Network Players has been set & tested for operation in a dedicated sub-network with a UnitiServer on the same switch. Which sort of explains some of the behaviour if the UPnP server is on a different or 'distance' sub-network. Hopefully my 5-port switch will arriving soon and I can adjust my setup, to reflect this configuration, and give my feedback.

Success (so far) - my reconfigured network (with RPi on a sub-network local to the ND5, with the NAS connected over the Mains plugs) is now transcoding FLAC to WAV, with the 24/192 files showing 100% of the Naim buffer at all times, once the initial UPnP 'Get' is invoked and serviced. There is some buffer drop at the end of a track, as the ND5 is preparing to play the next file.

When playing the 24/192 files, I see the 1.2 MB/s upstream from the RPi for the 9,216kb/s stream of uncompressed data, with the downstream I/O from the NAS peaking at 5.4MB/s when making the read over the CIFS mount via, with the CIFS read buffer set at 16M, so Read from the NAS only happens every 15s or so, over the 'Mains Plug' connection.
RPI CPU is around the 20% mark. This peaks at 100% when Asset runs the 'Detect New Tracks' but this doesn't affect the transcoding process.

Is the 'Detect New Tracks' running as a separate process? Could this run a lower priority to the main streaming process, or even only run when idle and not when playing.

Thanks for the support on this forum,
Simon.

[simes_pep]

Re: Asset UPnP for Raspberry pi

Success (so far) - my reconfigured network (with RPi on a sub-network local to the ND5, with the NAS connected over the Mains plugs) is now transcoding FLAC to WAV, with the 24/192 files showing 100% of the Naim buffer at all times, once the initial UPnP 'Get' is invoked and serviced. There is some buffer drop at the end of a track, as the ND5 is preparing to play the next file.

When playing the 24/192 files, I see the 1.2 MB/s upstream from the RPi for the 9,216kb/s stream of uncompressed data, with the downstream I/O from the NAS peaking at 5.4MB/s when making the read over the CIFS mount via, with the CIFS read buffer set at 16M, so Read from the NAS only happens every 15s or so, over the 'Mains Plug' connection.
RPI CPU is around the 20% mark. This peaks at 100% when Asset runs the 'Detect New Tracks' but this doesn't affect the transcoding process.

Is the 'Detect New Tracks' running as a separate process? Could this run a lower priority to the main streaming process, or even only run when idle and not when playing.

Thanks for the support on this forum,
Simon.

Yes, with the CIFS Read Buffer at the max 16M, there can be some Naim buffer exhaustion and dropout at the end of playing a transcoded 24/192 track when the ND5 makes the request to 'Get' the next track in the playlist, to ensure that there is 'Gapless' playback. This happens about 30s-40s before the end of the current track. The RPi is still processing the current buffer with the current track. So I have gone back to the default CIFS Read buffer size and everything plays fine during the track and at the end. Obviously there is more of a continuous stream of CIFS reads from the RPi to the NAS, with only a 60K buffer, but this seems to be fine over the 'Ethernet over Mains' connection.
So the 'take away' from this is, if you have to use 'Mains Plugs' because you are unable to have a hardwired Ethernet connection between NAS, UPnP Server and Network Player, make it between the UPnP Server and the NAS, and not the Network Player and the UPnP Server.

Network Player <---Hardwired Ethernet---> Asset UPnP on RPi <--Ethernet over Mains--> NAS over
Network Player <--Ethernet over Mains---> Asset UPnP on RPi <--Hardwired Ethernet---> NAS

Thanks,
Simon.

[Spoon]

Re: Asset UPnP for Raspberry pi

Update 21st January 2014

- Added webconfig page for browse tree
- Fixed internet radio not working
- Stability improvements
- Fixed playlist handling- now paths beginning with / are presumed absolute

[Batleys]

Re: Asset UPnP for Raspberry pi

Spoon, great stuff thanks.
Simon