A monitor for wireless internets that use BGP as their inter-AS Routing Protocol





A monitor for wireless internets that use BGP as their inter-AS Routing Protocol



Background and the AWMN model

In the AWMN and other community owned wireless internets network enthusiasts own, setup ,and maintain network and RF equipment on a Best Effort Basis. Most wireless nodes are Autonomous Systems (AS) and the vast majority of inter-AS links are wireless.

Most of the networking and RF equipment are exposed to harsh environments. Equipment failures, configuration errors , equipment incompatibilities , and plain old pranks are much much more common than in professional ISP networks.

There is a Number and Name Authority aka Host-master per community but there is not one authority that has administrative access to all the networking equipment. Usually there are Node Databases but when they exist they are incomplete and contain a large amount of false information. Many wireless communities in Greece are interconnected through wireless and Internet connections.

There are not peering agreements between nodes and communities and there are not established protocols to deal with problems. Community spirit and good interpersonal relationships between the node operators are not always enough. The vast coverage areas and the large number of network nodes increase the complexity of monitoring.





Abstract

This post discusses a Monitor for wireless internets that use BGP as their inter-AS Routing Protocol. The Monitor has access to the Node Database, BGP daemon tables, Routers, and Sensors across the internet. The Monitor alarms problems to Number and Name Authorities and Node Operators, draws a near real time connectivity graph and potentially automatically acts upon events to correct or contain problems.





Definitions

Node Database

A Database with IP Numbers, Node geographical coordinates (latitude,longitude,elevation), Node Operator Contact Information, DNS zones Link information and information for services provided by various nodes across the internet. The Number and Names Authorities aka Host-masters of the internet allocate and assign resources through the Node Database and most information is put there by the node operators. In the AWMN the Wireless Node Database WiND is used.



Node

An Autonomous System as defined in RFC 4271 where the most inter-AS links are wireless and most links within the Autonomous System are wired.



Node Operator

A community member who acts as a node administrator to one or more nodes. He usually has one wireless node on the rooftop of his residence.





1 Border Gateway Protocol Monitor

Our internet monitoring system has access to many BGP Speakers across the internet. In a wireless not-professional, not-homogeneous internet as-paths change very often and it is very common to find wrong as-paths, ghost links and ghost prefixes in a BGP table. We have access to many BGP tables but we need a way to attach levels of certainty to the information they contain. Also in community based internets configuration mistakes and pranks such as IP hijacks are common. The following procedure attempts to deal with ghost links, ghost prefixes, configuration mistakes, and IP hijackings by using the information on BGP tables from across the internet.




We get the BGP tables from across the internet in short regular intervals eg: 30 minutes

We detect Prepends log them and filter them out from the input to the rest of the process

We detect BGP Communities log them and filter them out from the input to the rest of the process

Now every path shorter than 255 ASes that contains an Autonomous System many times indicates a Loop and it should be alarmed and excluded from the following process. RFC 4271 25.1

We split the table in pairs of AS numbers to get the links.

To attach a level of certainty to a pair we calculate a weight by adding a point on every hop starting from the table in which we found it.
eg: consider the following as-path on the AS 1
2 3 4 5 6
the weight w of the pair 1 2 is equal to 0+1=1 w(1,2)=1 and w(2,3)=3 the smaller the weight is the most probable is that the link exists When an AS number x has a w(x,y)=1 , meaning we have access to its table a pair x,z with w(x,z)=33 is invalid and it should be alarmed.

For every prefix in the tables we check if it is assigned by the Number Authority and if not we alarm the host-master.

For every prefix in the tables we check if it is assigned by the Number Authority to the AS advertising it and if not we alarm the host-master.

If we find an AS not assigned to anyone by the Number Authority we alarm the host-masters.

We detect prefixes announced multiple times and according to our Number Authority should not be Anycast If we find such prefixes we can figure out using information from the Number Authority who is cheating or messed up and alarm host-masters and operators.

If we see a path leading to a node-AS where we have access to a BGP table with a prefix not announced there, then this is a Ghost Prefix this is the easy part. To Guess Ghost Prefixes with paths leading to Autonomous Systems in which we do not have access we will use the weights to attach certainty.

For the prefix weight we use the size of the as-path.

Again, the smaller the weight, the higher the probability the prefix exists.

While we traverse the tables lower weights replace higher weights.

We are trying to figure out if the small prefixes announced are valid. (An easy way to do this would be to check against a list maintained by the Number Authority) Invalid small prefixes can create parallel internet spaces for large prefixes announced by ASes with one or a few not stable links.

Detect prefixes that should not pass the BGP filters agreed by the community.



2 Classic and Not So classic Nagios Stuff

We will need to complement the Monitor with Classic nagios ICMP echo stuff since BGP tables may quickly become nothing more than tables of forwarding intentions in a wireless internet.

We could make use of tracepath scripts to detect asymmetries.

Since this is a wireless internet we will need to monitor Connection Quality CCQ, signal/noise ratios,noise level,and signal strength. These are available through SNMP in many routers so adding them to a nagios like system is simple ( examples )





3 Smart Routers

Most wireless routers now days are harsh embedded systems that have plenty of CPU and RAM. We may use scripts to add some brains and network healing capabilities to them.

eg: In the AWMN is common enough for a dish antenna to move by strong winds or a feeder fill with water after a storm. The link quality may drop drastically but the BGP session remains active. However, since the link quality is very poor many frames and IP packets drop, the ones using it suffer, and the quality of the internet drops. Another as long or even longer but higher in quality route could be used instead. A script may monitor the signal strength of each link and close the appropriate BGP session if the link quality drops drastically (an example script)

A smart router may prepepend its autonomous system to the AS-path as a first measure and close the BGP session as a last measure. The router could also act based on Connection Quality and other link quality metrics.









URI: A monitor for wireless internets that use BGP as their inter-AS Routing Protocol

WTF Google Safe Browsing

WTF Google Safe Browsing?

safebrowsing ipduh.com

Truth

IPduh is not compromised and has nothing to do with malware distribution.
Quite the opposite is True.



Rant

Google Safe Browsing thank you for waisting my time, the headache and this post.



What Happened

ipduh has a little simple tool called bouncer that checks if a URI is mentioned in a black list before forwarding you there. It is used mostly to hide referrals by people who post links and it uses --among other external and ipduh tools-- the Google Safe Browsing API ( when it works ) --what an irony! ( on multiple levels )

So, Safe Browsing saw uris like the following and made some false associations.

http://ipduh.com/url/bouncer/?fitmaster.gr
http://ipduh.com/url/bouncer/?antarsya.gr
http://ipduh.com/url/bouncer/?grayicon.com


According to Google Safe Browsing those sites were distributing malware at some time.
For most of the sites above the bouncer was saying:
Attention!
malware
Please check dns/bl


Still want to go? Use: www





A Few Pointers to the Safe Browsing People

  • You are not the only one who may warn about potentially malicious websites.
  • When you say that x site is very bad be sure about it.
  • Vague Scary Words do not help anyone.
  • If you continue using URI associations like that you are going to become a First Grade Annoyance Tool that can easily be manipulated by clueless haxors with one week of training.




WTF Google "Safe" Browsing

Enable - Disable BGP peer based on signal strength.

In the AWMN is common enough for a dish antenna to move from strong winds or for a feeder to fill with water after a storm. The link quality may drop drastically but the BGP session remains active. However, since the link quality is very poor many frames drop, the people using it suffer and the quality of the internet drops. Another as long or even longer but higher in quality route could be used instead. A script may monitor the signal strength of each link and close the appropriate BGP session if the link quality drops drastically.

The following is such a script written for RouterOS by George Katsimagklis aka SV1BDS (Ham Radio Cosign and awmn handle) inspired by an AWMN VoIP conversation we had. There is a thread in Greek about it the awmn forum.

The script

# monitor WLAN1 status
:global wlan1status
:global wlan1RSSI
:local interfacename "wlan1-xyz"
:local myemail "69xxxxxxxx@mycosmos.gr,zzzzz@yahoo.com"
:local RSSIlevel -75
:local SMTPserver "smtp.gmail.com"
:local peername "SV1XYZ"

# Local variables
:local int
:local mysubject
:local RSSIs "-1dB"
:local RSSI
:local statusnow "down"
:local s

:foreach int in=[/interface wireless registration-table find interface=$interfacename] do={
    :set statusnow "up"
    :set RSSIs [/interface wireless registration-table get $int signal-strength]}

:set RSSI [:tonum [:pick $RSSIs 0 [:find $RSSIs "d" 0]]]

#:log info $RSSI
#:log info $statusnow

:if ([:typeof $wlan1status] = "nothing") do={
     :set wlan1status $statusnow
  } else={:if ($wlan1status != $statusnow) do={
              :set mysubject ($interfacename." on ".[/system identity get name]." is ".$statusnow)
             /tool e-mail set address=[:resolve $SMTPserver]
             :foreach s in=[:toarray $myemail] do={
                  /tool e-mail send to=$s subject=$mysubject}
             :log info $mysubject
             :set wlan1status $statusnow
             :foreach int in=[/routing bgp peer find name=$peername] do={
                 :if ($statusnow = "up") do={
                      /routing bgp peer enable $int
                 } else={
                      /routing bgp peer disable $int
                 }
             }
         } 
 }
:if ($statusnow="up") do={
  :if ([:typeof $wlan1RSSI] = "nothing") do={
      :if ( $RSSI<$RSSIlevel) do={
          :set wlan1RSSI "BAD"
          :set mysubject ($interfacename." on ".[/system identity get name]." RSSI is ".$RSSI." (".$RSSIlevel.")")
          /tool e-mail set address=[:resolve $SMTPserver]
          :foreach s in=[:toarray $myemail] do={
              /tool e-mail send to=$s subject=$mysubject}
          :log info $mysubject
          :foreach int in=[/routing bgp peer find name=$peername] do={ /routing bgp peer disable $int }
      } else={
           :set wlan1RSSI "OK"
           :foreach int in=[/routing bgp peer find name=$peername] do={ /routing bgp peer enable $int }
      }
  } else={:if ( $RSSI<$RSSIlevel and $wlan1RSSI="OK") do={
                 :set wlan1RSSI "BAD"
                 :set mysubject ($interfacename." on ".[/system identity get name]." RSSI is ".$RSSI." (".$RSSIlevel.")")
                 /tool e-mail set address=[:resolve $SMTPserver]
                 :foreach s in=[:toarray $myemail] do={
                       /tool e-mail send to=$s subject=$mysubject}
                 :log info $mysubject
                 :foreach int in=[/routing bgp peer find name=$peername] do={ /routing bgp peer disable $int }
                 }
              :if ( !($RSSI<$RSSIlevel) and $wlan1RSSI="BAD") do={
                 :set wlan1RSSI "OK"
                 :set mysubject ($interfacename." on ".[/system identity get name]." RSSI is ".$RSSI." (".$RSSIlevel.")")
                 /tool e-mail set address=[:resolve $SMTPserver]
                 :foreach s in=[:toarray $myemail] do={
                        /tool e-mail send to=$s subject=$mysubject}
                 :log info $mysubject
                 :foreach int in=[/routing bgp peer find name=$peername] do={ /routing bgp peer enable $int }
               }
  }
}
#:log info "end"






Enable-Disable BGP peering based on signal strength

5GHz - 3x3 MIMO Antenna - 3x3 MIMO Feeder

In the past three months me, nikolas_350, and other members of the AWMN community have been trying to design Antennas for Long Range 3x3 Multiple Input Multiple Output MIMO Wireless links.

When Atheros based 3x3 MIMO miniPCI express cards were made available at reasonable prices we decided to come up with a 3x3 MIMO 5GHz feeder design or at least a practical 3x3 MIMO Antenna System and after one, heated at times, discussion we concluded to construct three types of Feeders and run a series of tests. We chose to try out combinations of polarizations , single & multiple offset dish antennas and feeders , and LMR cable lengths until we achieve a practical way to differentiate the three streams enough to achieve 3x3 MIMO above MCS20 on long range links.

We started our quest confident that we can create a four dish-four feeders per link system, hopping that in the process we will come up with a practical two dish-two feeders per link system.

In the AWMN we use mostly offset dish Antennas with 5GHz copper feeders we make ourselves.



We have single ( one polarization ) feeders and double ( horizontal and vertical polarization ) feeders. Our double feeders have been working great in Long 2x2 MIMO 802.11N Links.

classic AWMN 5GHz Feeders







We decided to use RouterOS powered devices in our tests since many people in this group were familiar only with this operating system. Also we had heard rumors that RouterOS works OK with the Atheros based 3x3 MIMO miniPCI express cards and we wanted to try out the Nv2 protocol with 3x3 MIMO.



The Feeders we constructed


The tough parts were made by nvac. We are not a factory --yet.

The 0,90,225 feeder

The 0,120,240 degrees feeder from behind


The 0,120,240 feeder
The 0,120,240 degrees feeder without n-types from above



The 0,90,270 feeder



Our Tests

Our tests were simple.
It works means three spatial streams registered with a sustainable index above MCS20 and bandwidth throughput greater than the 2x2 MIMO upper limit. The 2x2 MIMO maximum bandwidth throughput is approximately 210Mb/s UDP for a 40MHz channel.



Test 1)
Indoor
Three small antennas on each node
A few centimeters up to 10 meters distance between the nodes

When I say small, I mean it, the antennas were λ/4 copper wires.

In the few centimeters up to 2 meters distances the results were excellent and we saw the wireless registration locking at 450Mb/s, and bandwidth tests reporting more than 300Mb/s UDP traffic.





Test 2)
Indoor
One 0,90,225 feeder on each node
5m and 10m distance between the nodes
In a nutshell, it did not work.
We saw momentarily rates above the MCS20 index.



Test 3)
Indoor
One 0,120,240 feeder on each node
5m and 10m distance between the nodes
In a nutshell, it did not work.
We saw momentarily rates above the MCS20 index.



Test 4)
Indoor
One 0,90,270 feeder on each node
It did not work.



Test 5)
Indoor
Two Feeders on each node
One feeder with horizontal and vertical polarization and one feeder with diagonal polarization
It works.
However, we are indoors and away from practical.



The Long Outdoor Tests were made in the ipduh-nikolasc link ~3,5 Km distance.

The offset parabolic dish we are using are 80cm Gilbertini.

Test 6)
Outdoor
One offset parabolic dish with a 0,90,225 feeder on each node
It does not work.
We saw momentarily rates above the MCS20 index.



Test 7)
Outdoor
One offset parabolic dish with a 0,90,270 feeder on each node
It does not work.



Test 8)
Outdoor
One offset parabolic dish with a 0,120,240 feeder on each node
It does not work.
We saw momentarily rates above the MCS20 index.



Test 9)
Outdoor
Two offset parabolic dish with two feeders on each node
One Feeder with vertical & horizontal polarizations and the other with a diagonal 45 degree polarization.
We did not test it yet.


Test 10)
Outdoor
1 offset parabolic dish with the 0,90,225 feeder on each side and variable length LMR cables.
We may achieve to differentiate one stream enough this way.
We did not test it yet.


Test 11)
Outdoor
1 offset parabolic dish with the 0,120,240 feeder on each side and variable length LMR cables.
We did not test it yet.




Waiting for nikolas_350 to recover from the Summer Vacation and get back to business.


to be continued ...



5GHz - 3x3 MIMO Antenna - 3x3 MIMO Feeder

Monitoring wireless BGP internets - Part 3

This is a rough sketch of a network monitor for community owned wireless BGP internets.

Consider a BGP internet monitoring system with access to
many BGP tables across the internet
and a Number Authority-Node Database eg: WiND or NodeDB

I am trying to figure out a process to attach levels of certainty to the information in the tables, find ghost links and ghost prefixes, and throw alarms.

The process aka the order algorithm:

We detect and filter out Prepends

Now, every path that contains an Autonomous System many times indicates a Loop and it should be alarmed and excluded from the following process.

Links: We split the table in pairs of AS numbers to get the links.

To attach a level of certainty to a pair we calculate a weight by adding a point for every hop starting from the AS in which we found the pair.
eg: consider the following as-path on the AS 1
2 3 4 5 6
the weight w of the pair 1 2 is equal to 0+1=1 w(1,2)=1 and w(2,3)=3 the smaller the weight is the most probable is that it exists

While we traverse the BGP Tables lower weights replace higher weights.

When, for an AS number x there is a w(x,y)=1 ( we have access to its table )
a pair x,z with w(x,z)=33 is invalid for sure and it should be alarmed.

Prefixes:

For every prefix in the tables we check if it is assigned by the Number Authority, if not we alarm.

For every prefix in the tables we check if it is advertized by the AS in which is assigned by the Number Authority, if not we alarm.

We detect prefixes announced multiple times and according to our Number Authority should not be Anycast If we find such prefixes we use the information from the Number Authority to figure out who is cheating or messed up to alarm.

Ghost Prefixes:

If we see a path leading to a node in which we have access with a prefix not announced there, then this is a Ghost Prefix. This is the easy part. We may attempt to guess Ghost Prefixes with paths leading to Autonomous Systems in which we do not have access using weights.

For the prefix weight we use the size of the as-path.
Again, the smaller the weight, the higher the probability the prefix exists.

Again, while we traverse the tables lower weights replace higher weights.





This BGP monitoring system combined with
  • information from a Node Database eg: WiND or NodeDB
  • some classic nagios stuff
  • and some not so classic nagios stuff ( nagios1 , nagios2-missinglink router_scripts-missinglink )
can create monitoring systems that can throw meaningful alarms, draw near real time maps, and even do some shelf-healing.



thoughts on monitoring wirelless BGP internets