Planet Catalyst

23 January 2012

Dan Marsden

Meta-course enrolments and failed Moodle 2 upgrades oh my!

We just upgraded one of our clients sites that makes extensive use of the meta-course enrolment feature combined with ‘nosyncroleids’ from 1.9 to 2.2 – they use meta-courses to enrol their students but use manual enrolments for their teachers so the teacher role was set in ‘nosyncroleids’ – during the upgrade these manual enrolments for teachers seem to have been corrupted – after upgrade the users show as “enrolled” using meta-enrolment but don’t have any roles in their courses. This seems strange as in the 1.9 site they were “manual” enrolments.

It looks like Petr Skoda may have fixed this as part of MDL-29684 but this is only fixed in master, not the stable branches. (and I haven’t tested it)

And unfortunately as the sync process has already screwed up the enrolments on the courses, simply applying the patch to the clients 2.2 site wouldn’t work – the only way it seems it might work would be to revert back to the 1.9 site and re-run the upgrade with the patch in place – or manually go through each course and fix the enrolments. Unfortunately we didn’t notice this early enough and the site was already being used so the client is now going through each course and manually fixing the enrolments for their teachers (time-consuming and frustrating!)

IMO this sort of issue should be back-ported but hopefully this post might help prevent others from experiencing the same pain!

by dan at 23 January 2012 23:33

14 January 2012

Francois Marier

Debugging OpenWRT routers by shipping logs to a remote syslog server

Trying to debug problems with consumer-grade routers is notoriously difficult due to a lack of decent debugging information. It's quite hard to know what's going on without at least a few good error messages.

Here is how I made my OpenWRT-based Gargoyle router send its log messages to a network server running rsyslog.

Server Configuration

Given that the router (192.168.1.1) will be sending its log messages on UDP port 514, I started by opening that port in my firewall:

iptables -A INPUT -s 192.168.1.1 -p udp --dport 514 -j ACCEPT

Then I enabled the UDP module for rsyslog and redirected messages to a separate log file (so that it doesn't fill up /var/log/syslog) by putting the following (a modified version of these instructions) in /etc/rsyslog.d/10-gargoyle-router.conf:

$ModLoad imudp
$UDPServerRun 514
:fromhost-ip, isequal, "192.168.1.1" /var/log/gargoyle-router.log
& ~

The name of the file is important because this configuration snipet needs to be loaded before the directive which writes to /var/log/syslog for the discard statement (the "& ~" line) to work correctly.

Router Configuration

Finally, I followed the instructions on the Gargoyle wiki to get the router to forward its log messages to my server (192.168.1.2).

After logging into the router via ssh, I ran the following commands:

uci set system.@system[0].log_ip=192.168.1.2
uci set system.@system[0].conloglevel=7
uci commit

before rebooting the router.

Now whenever I have to troubleshoot network problems, I can keep a terminal open on my server and get some visibility on what the router is doing:

tail -f /var/log/gargoyle-router.log

by François Marier (noreply@blogger.com) at 14 January 2012 22:55

12 January 2012

Timothy Goddard

Messing around dealing with random packet loss

One of my interests recently has been around network quality and how well protocols such as TCP cope with noise. Having just spent a week trying to use internet in various backpackers, network quality has come to the front of my brain again.

When a router can’t forward traffic as quickly as it’s getting it, it starts dropping packets. TCP takes advantage of this to measure link capacity – it increases the amount of traffic being sent until it starts losing packets then reduces the speed again. This generally works very well, but has an issue with packet loss not related to the data being sent. Many networks just have noise or lose packets randomly, and TCP will tend to back right off to a trickle when this happens completely unnecessarily.

I’ve been having a play with some simple erasure correction over networks. The result so far is a messy ruby script which will set up a tunnel with some erasure tolerance between linux boxes. EDIT: script updated. Was insanely buggy and kept losing link when client changed source port. Now repairs connection when that happens, editing this now over the link.. It breaks each packet it gets on the tun interface in to 3 UDP packets, and can reassemble any 2 of those in to the original. It takes just over half as much again traffic (adds some headers) but is a bit more resilient to random packet loss.

You can set up a server instance like:

you@server$ sudo ruby fectun.rb 0.0.0.0 4943 server PASSWORD

It will bind to the IP and port given and wait for a client to connect. The word “server” is magic – no decent command line options yet. The password should be chosen and kept the same between them – the server can only handle one client at once and any client with the password can take over the connection.

To connect with a client:

you@client$ sudo ruby fectun.rb 0.0.0.0 9000 my.other.computer 4943 PASSWORD

If all goes well, the server end should tell you it’s switching its destination to the client that just connected. The port may be mangled if the client is behind a NAT router.

Each end now has a “tun” interface connected to each other. These need to be given IP addresses:

you@server$ sudo ifconfig tun0 10.93.0.1/24

you@client$ sudo ifconfig tun0 10.93.0.2/24

Then from the client you should be able to:

you@client$ ping 10.93.0.1

or set up a SOCKS proxy by running SSH over the link:

you@client$ ssh you@10.0.0.1 -D 8080

It’s all a little sketchy at the moment but have managed to get a link to my server going from home. I hope to extend this to batch packets together within the erasure code so that it doesn’t send 3 times as many packets.

P.S. Please use this nicely. Could be quite anti-social on busy networks if you sent a lot of traffic over it.

12 January 2012 13:57

Last updated

27 January 2012 20:30
All times are UTC.

Disclaimer

Planet Catalyst is an aggregation of public weblogs written by Catalyst IT employees. The opinions expressed in these weblogs and hence this aggregation are those of the original authors.

Planet Catalyst is not a product or publication of Catalyst IT; as such, it does not necessarily represent the views of Catalyst IT.

If you want to be added, please check the Planet Catalyst page on the internal wiki.


Planet Catalyst


23 January 2012 Dan Marsden Meta-course enrolments and failed Moodle 2 upgrades oh my! We just upgraded one of our clients sites that makes extensive use of the meta-course enrolment feature combined with ‘nosyncroleids’ from 1.9 to 2.2 – they use meta-courses to enrol their students but use manual enrolments for their teachers so the teacher role was set in ‘nosyncroleids’ – during the upgrade these manual enrolments for teachers seem to have been corrupted – after upgrade the users show as “enrolled” using meta-enrolment but don’t have any roles in their courses. This seems strange as in the 1.9 site they were “manual” enrolments. It looks like Petr Skoda may have fixed this as part of MDL-29684 but this is only fixed in master, not the stable branches. (and I haven’t tested it) And unfortunately as the sync process has already screwed up the enrolments on the courses, simply applying the patch to the clients 2.2 site wouldn’t work – the only way it seems it might work would be to revert back to the 1.9 site and re-run the upgrade with the patch in place – or manually go through each course and fix the enrolments. Unfortunately we didn’t notice this early enough and the site was already being used so the client is now going through each course and manually fixing the enrolments for their teachers (time-consuming and frustrating!) IMO this sort of issue should be back-ported but hopefully this post might help prevent others from experiencing the same pain! by dan at 23 January 2012 23:33 14 January 2012 Francois Marier Debugging OpenWRT routers by shipping logs to a remote syslog server Trying to debug problems with consumer-grade routers is notoriously difficult due to a lack of decent debugging information. It's quite hard to know what's going on without at least a few good error messages. Here is how I made my OpenWRT-based Gargoyle router send its log messages to a network server running rsyslog. Server Configuration Given that the router (`192.168.1.1`) will be sending its log messages on UDP port 514, I started by opening that port in my firewall: iptables -A INPUT -s 192.168.1.1 -p udp --dport 514 -j ACCEPT Then I enabled the UDP module for rsyslog and redirected messages to a separate log file (so that it doesn't fill up `/var/log/syslog`) by putting the following (a modified version of these instructions) in `/etc/rsyslog.d/10-gargoyle-router.conf`: $ModLoad imudp $UDPServerRun 514 :fromhost-ip, isequal, "192.168.1.1" /var/log/gargoyle-router.log & ~ The name of the file is important because this configuration snipet needs to be loaded before the directive which writes to `/var/log/syslog` for the discard statement (the "& ~" line) to work correctly. Router Configuration Finally, I followed the instructions on the Gargoyle wiki to get the router to forward its log messages to my server (`192.168.1.2`). After logging into the router via ssh, I ran the following commands: uci set system.@system[0].log_ip=192.168.1.2 uci set system.@system[0].conloglevel=7 uci commit before rebooting the router. Now whenever I have to troubleshoot network problems, I can keep a terminal open on my server and get some visibility on what the router is doing: tail -f /var/log/gargoyle-router.log by François Marier (noreply@blogger.com) at 14 January 2012 22:55 12 January 2012 Timothy Goddard Messing around dealing with random packet loss One of my interests recently has been around network quality and how well protocols such as TCP cope with noise. Having just spent a week trying to use internet in various backpackers, network quality has come to the front of my brain again. When a router can’t forward traffic as quickly as it’s getting it, it starts dropping packets. TCP takes advantage of this to measure link capacity – it increases the amount of traffic being sent until it starts losing packets then reduces the speed again. This generally works very well, but has an issue with packet loss not related to the data being sent. Many networks just have noise or lose packets randomly, and TCP will tend to back right off to a trickle when this happens completely unnecessarily. I’ve been having a play with some simple erasure correction over networks. The result so far is a messy ruby script which will set up a tunnel with some erasure tolerance between linux boxes. EDIT: script updated. Was insanely buggy and kept losing link when client changed source port. Now repairs connection when that happens, editing this now over the link.. It breaks each packet it gets on the tun interface in to 3 UDP packets, and can reassemble any 2 of those in to the original. It takes just over half as much again traffic (adds some headers) but is a bit more resilient to random packet loss. You can set up a server instance like: you@server$ sudo ruby fectun.rb 0.0.0.0 4943 server PASSWORD It will bind to the IP and port given and wait for a client to connect. The word “server” is magic – no decent command line options yet. The password should be chosen and kept the same between them – the server can only handle one client at once and any client with the password can take over the connection. To connect with a client: you@client$ sudo ruby fectun.rb 0.0.0.0 9000 my.other.computer 4943 PASSWORD If all goes well, the server end should tell you it’s switching its destination to the client that just connected. The port may be mangled if the client is behind a NAT router. Each end now has a “tun” interface connected to each other. These need to be given IP addresses: you@server$ sudo ifconfig tun0 10.93.0.1/24 you@client$ sudo ifconfig tun0 10.93.0.2/24 Then from the client you should be able to: you@client$ ping 10.93.0.1 or set up a SOCKS proxy by running SSH over the link: you@client$ ssh you@10.0.0.1 -D 8080 It’s all a little sketchy at the moment but have managed to get a link to my server going from home. I hope to extend this to batch packets together within the erasure code so that it doesn’t send 3 times as many packets. P.S. Please use this nicely. Could be quite anti-social on busy networks if you sent a lot of traffic over it. 12 January 2012 13:57	Last updated 27 January 2012 20:30 All times are UTC. Disclaimer Planet Catalyst is an aggregation of public weblogs written by Catalyst IT employees. The opinions expressed in these weblogs and hence this aggregation are those of the original authors. Planet Catalyst is not a product or publication of Catalyst IT; as such, it does not necessarily represent the views of Catalyst IT. If you want to be added, please check the Planet Catalyst page on the internal wiki. Subscriptions Andrew Caudwell Andrew Ruthven Brett Wilkins Chris Cormack Dan Marsden Donovan Jones Edward Murrell Francois Marier Jonathan Harker Josh Waihi Kristina Hoeppner Martyn Smith Matthew Holloway Pass the Source Piers Harding Russel Garlick Sibylle Schwarz Timothy Goddard