Category Archives: Reference

Producing Pretty Graphs

My aim to reproduce Figure 1: Internet users and non-users by age group (years), 2012 Q3 from the Internet Access Quarterly Update, Q3 2012 from the Office for National Statistics. For this I will be using Octave

THE PLAN

1) Download data as xml and convert to csv
2) Read the data into Octave and check this it has been read in correctly
data = data = dlmread(“<my_file.csv>”,”,”);
size(data)
The expected output will be 8 4
3) Divide the date into the correct axis
x = data(:,1);
ya = data(:,2);
yb = data(:,3);
yc = data(:,4);
4) Plot the new data

     plot(x,ya,”;Used within last 3 months;”,x,yb,”;Used more than 3 months ago;”,x,yc,”;Never used;”)

5) Label the x and y axis
xlabel(“Age Group(Years)”);
ylabel(“Percentage who have used the internet”);
6) Give the graph a title

     title(“Internet users and non-users by age group (years), 2012 Q3”);

7) Output the graph and save
print(“test.pdf”,”-dpdf”);

LaTeX – The IEEE Surveys & Tutorials Way (Pt 2)

This the second article in a series on using the IEEEtrans and LaTeX to form a survey suitable for submission to IEEE surveys & tutorials

Yesterday’s article made use of the bare bones template for journals provided by IEEEtrans, now we are going to look at the requirements of a survey in IEEE surveys & tutorials

IEEE Surveys & Tutorials

The following information is largely taken from the Information for Authors page, on the IEEE Communications Surveys & Tutorials site.
The focus of this article is a survey not a tutorial, therefore all of the points below are regarding surveys and not necessary tutorials.

IEEE Communications Survey & Tutorials is a free online journal published by the IEEE Communications Society. Article appear in both the IEEE Xplore Electronic Library and the IEEE ComSoc Digital Library

To finding existing surveys to read and use as examples, you can sign up to a guest account with IEEE ComSoc Digital Library which will give you free access, at the time of writting, the IEEE ComSoc Digital Library website was down, so I’ve not yet been able to verify this.

IEEE Communications Surveys and Tutorials is targeted for the generalist throughout the field of communications and communications networking. Intended readers include those involved in research, development, deployment, or instruction in fields related to communications.

All surveys should be tutorial in nature and should be written in a style comprehensible to readers outside the speciality of the article.

Typically, mathematical equations should be kept to a minimum, and not be used unless they are vital to the presentation. Articles intended to instruct in a mathematical area should be presented at a level comprehensible to readers with appropriate backgrounds, and the appropriate background should be carefully delineated in the introductory section of the article.

The term survey, as applied here, is defined to mean a survey of the literature. A survey article in IEEE Communications Surveys & Tutorials should provide a comprehensive review of developments in a selected area, covering its development from its inception to its current state and beyond, and illustrating its development though liberal citations from the literature. Such citations would naturally lead to an authoritative and comprehensive bibliography. The objective is to provide the reader with a sense of the history, development, and future of the topic area, including its interactions with other areas, and enough information to comprehend the development and identify the major players.

As an example, the article “A survey of markup languages” might discuss a number of markup languages, such as WML, XML, HTML, CHTML, and voiceXML. The article might define the term “markup language” and describe some general features and objectives by way of introduction. The article might then provide a time-line of events leading to the advent of markup languages, citing major milestones and breakthroughs. From there, the article might describe the markup languages in chronological order, showing how previous languages developed from previous ones through liberal citations to the literature. The article might conclude by giving the author’s well-thought-out opinions on the future.

References must be numbered sequentially, not alphabetically. The basic reference format is: [#] L. Brakmo and L. Peterson, ” TCP Vegas: End to End Congestion Avoidance on a Global Internet,” IEEE JSAC, vol. 13, no. 8, Oct. 1995, pp. 1465-80.

Authors must clearly state the category of the article in the abstract and again in the introductory section and also clearly state the scope of the article. For example, there must be a statement of the form “This article surveys the literature over the period 1990-2001 on turbo codes as they apply to wireless communications.”

Authors are encouraged to consider inclusion of multimedia materials in cases where such material would substantially improve the value of the article to the reader. HOW ???

Figures and tables should be used liberally. 

There are no limits on paper length, number of figures, references, etc.

Required formats for electronic submission is PDF.
Submit survey’s using the LaTex structure. Include the compiled pdf, figures (as eps files with fonts embedded), bios and photos in the final article, keywords, and abstract
(Note: For the best presentation of your article’s entry in IEEE Xplore, do not include equations in the abstract.)

Once the survey is complete it’s submitted via the ManuscriptCentral website

LaTeX – The IEEE Surveys & Tutorials Way (Pt 1)

This article aims to give an overview of setting out a survey for submission to an IEEE Journal.

For this article, I will be using ubuntu 12.10, 32 Bit and Vim as my text editor.

IEEEtran is the offical LaTeX class for authors of IEEE transaction journals and coferences

My primary source of information for this article is the latest version of the IEEEtran package IEEEtrans director. Other helpful information is available here.

INSTALLATION & SETUP

The essential book on LaTeX

The IEEEtrans directory (as linked above) includes a “bare bones example of a IEEE journal, called bare_jrnl.tex. We will now use the IEEEtran LaTeX class to compile this .tex file into a pdf file.

To find out the location that we need to place IEEEtran.cls in, use:

$ locate article.cls

For me, the first location returned is /usr/share/texlive/texmf-dist/tex/latex/base/article.cls

I now need to move the IEEEtran.cls that I download into the same directory as article.cls. So move the the IEEEtrans directory and execute:

$ sudo cp  IEEEtran.cls

You can now tell Tex about this new LaTeX class using:

$ sudo texthash

Latex is already installed on my system but to allow us of the IEEEtran class, you may also need:

$ sudo apt-get install texlive-fonts-recommended

If you get an error of the form:

Font OT1/ptm/m/n/9=ptmr7t at 9.0pt not loadable: Metric (TFM) file not found
Then it is likely that you need to install the texlive-fonts-recommended package.
Now we are going to compile the bare_jrnl.tex, by moving to its directory and excauting:
$ latex bare_jrnl.tex
You can now view the outputted document using:
$ xdvi bare_jrnl
NEXT STEPS
Now you have a bare bones copy of your paper, you can customize it to meet your specific requirements and add the content of your paper. I will look at this in more detail tomorrow. is the Information for Authors

VIM – The CompSci Classic

Learning to escape VIM

I have a confession, the first I launched vim I could not even work out how to quit it so I took the lazy way out and just closed the terminal. This put me right off vim for quite a while. Now its time to really try to learn it.

Vim has two modes: insert and normal, when you launch vim it will go straight into normal mode, from normal mode type i to switch to insert mode and then press ESC to return back to normal mode.

GOLDEN SECRET 1: To exit VIM, enter :q in the normal mode

Normal mode basics 
You can delete the character under the cursor using x and using :w to save. To cut the current line use dd, this will delete the text on the current line and also delete the line itself, the cursor will move to the next line, which is now where the original line was. To copy the current line use yy. Now to paste the recently cut line, use p  to insert the text onto the line below where the cursor currently is or P to insert text before cursor.

Instead of the arrow keys, you can make use of hjkl to move the cursor, but personal I prefer to use the arrow keys

Entering Insert mode
So far we have entered insert mode using i, there are some alternative ways to enter insert mode, which allow you to move to insert mode and perform a useful operation in one key stoke, some of the alternative methods to enter insert mode are:

  • a – insert mode and move cursor back one charactor
  • o – insert mode and insert a new line after the current one
  • O – insert mode and insert a new line before the current one
  • cw – insert mode and delete the current word
GOLDEN SECRET 2: Insert mode has auto complete, use Ctrl-n to activate

Moving around normal mode
As well as using the arrow keys (or hjkl) to navigate around normal mode, there are plenty of useful extra shortcuts for navigating around. 0 jumps to first character in the row and $ goes to the last character in the row. ^ jumps to the first non-blank character of a line and g_  goes to the last non_blank character of line.

To search for a particular work or character, use / and enter search term. This will move the cursor to the next occurrence of the search term after the cursor, so to search for the first occurrence of a term in a file, put the cursor at the start of the file and then use / to search for the term.

To jump to the start of the next word use w and to go to end of this word use e, when  a word is composed of letters and underscores only, if you want to word to include special characters use W and E instead

GOLDEN SECRET 3: % means go to corresponding (, { and [

Text editor basics
u is used to undo and Crtl-r to redo. To open a file use :e , :w to save, :saveas , 😡 to save and quit. :q! to quit without saving. . will repeat last   command and N will repeat the command N number of times. NG goes to line N and G is go to last line

 
 
GOLDEN SECRET 4: * means to next occurrence of word under cursor and # means go to previous

TCPtrace – An introduction

What is TCPtrace ?

Wireshark wins over TCPtrace on GUI  

Its a tool designed to analyze the output logs from TCPdump. Previously, in my introduction to TCPdump I highlighted that the output logs created by TCPdump were not plain text and only special programs could interpret them, TCPdump is one of these program, as is Wireshark and TCPtrace.

So TCPtrace takes the output file from TCPdump as an input and it then outputs useful information and graphs.

How do I get TCPtrace ?

I downloaded it from the Ubuntu repositories using the typical ‘sudo apt-get install tcptrace’. If this is not possible you can download it from here.

How do I input a TCPdump file to TCPtrace ?

You can call TCPtrace with a TCPdump file using  ‘tcptrace ‘ where my-file is the name of the file outputted by TCPdump. For example you could do something like:
$ sudo tcpdump -v -i wlan0 -w my_tcpdump_output -c 100
$ tcptrace my_tcpdump_output

The above will run TCPdump and create the output file called “my_tcpdump_output”, this file is then passed as a argument to the TCPtrace tool

How do i interpret the basic TCPtrace output ?

The structure of the output is (in order from the top) :

  • 1st line returns the name of the TCPdump output file that TCPtrace is currently analyzing
  • Then it printed the version of TCPtrace and then this version was last compiled
  • The next line states the number of packets seen and the number of those which were TCP
  • The following line gives the elapsed wallclock time, this is the time TCPtrace took to process the file and it then gives the average number of packets processed by TCPtrace per second
  • The following line gives the trace file elasped time, this is the time between the capture of the first packet and the last packet in the file
  • The sequential lines contain information on each TCP connection
    • First it gives the address and ports of the two machines involved in the connection
    • Then is the label given to this connection by TCPtrace is printed in parenthesis
    • The number proceeding ‘>’ is the number of packets seen in the first host to second host direction
    • The number proceeding ‘<‘ is the number of packets seen in the second host to the first host direction
    • Then the connection is labelled with ‘(complete)’ or ‘(reset)’, with the connection being labelled as complete if SYN and FIN packets were seen

This output is TCPtrace’s brief output. Just like TCPdump, you can stop the translation of IP address to domain names using the ‘-n’ opinion.

When using TCPdump, you can see more detailed output using the ‘-v’ option but with TCPtrace you can see more detailed output using the ‘-l’ option.

When adding options to TCPtrace, you need to ensure the you place the extra options before the name of the input file and after the tool name.

When viewing the output from the long mode (when -l is the option) then all information is labelled. I’m now going to explain each label given in long output (warning .. this might take a while):

Packets and ACKS

  • total packets – number of packets sent in that specific direction
  • ack pkts sent – number of ACKs sent in that direction
  • pure acks sent – number of ACK sent without data and the SYN,FIN&RST not set
  • sack pkts sent – number of selective ACKs sent in that direction
  • dsack pkts sent – number of duplicate selective ACKs sent in that direction
  • max sack blks/ack – maximum number of selective ACK blocks seen in any selective ACK packet

Retransmissions

  • unique bytes sent – total number of bytes sent excluding retransmittions
  • actual data pkts – total number of packet with at least 1 byte of TCP payload
  • actual data bytes – total number of bytes seen including retransmittion
  • rexmt data pkts – total number of packets that where retransmittions
  • rexmt data bytes – total number of bytes of data that where retransmittions

Window scaling / Probing

  • zxnd probe pkts – total number of window probe packets seen
  • zxnd probe bytes – total number of bytes sent in window probe packets
  • outoforder pkts – number of packets that arrived out of order
  • pushed data pkts – number of packets with the PUSH bit set, this means that the buffered data should be sent to the receiving application
  • SYN/FIN pkts sent – number of packets with SYN or FIN bits set

etc… (sorry I hate leaving things half done, but I really wanted to move on, its in my to-do list)

How do I get RTT (Round-Trip-Time) from TCPtrace ?

TCPtrace will generate statistics on RRT when using with the opinions ‘-r’ and ‘-l’. This will give data on RRT including the number of RTT samples found, RTT minimum,RTT maximum, RTT average, RTT standard deviation, RTT from TCP’s hand shake. The same data is then available again for full-sized RTT samples only.

TCPdump – An introduction

As per usual, if you find any mistakes, concepts that could be explained better or have something to add, then please comment below and I’ll correct it.

What is TCPdump ?

TCPdump is a command line tool that is used to look at packets on the network. TCPdump does not generate the packets itself but instead it analyzes the packets generated by other applications and from this, it can determine network behaviour and performance.
Despite being called TCPdump, you can choose from a range of protocols including IP,ARP, TCP and UDP.

How do I get TCPdump ?

From Ubuntu, I simply got it via ‘sudo apt-get install tcpdump’. Otherwise the public repo is located here, along side some useful information and the manual pages.
You can also check the dependencies of TCPdump using ‘apt-cache depends tcpdump’. The output that this returns for me is:
  Depends: libc6
  Depends: libpcap0.8
  Depends: libssl1.0.0
  Suggests: apparmor
  Conflicts: tcpdump:i386
If you get the package for the public repo, then you need will extract the content of the .tar.gz file using ‘tar -zxf tcpdump-4.3.0.tar.gz’ after using cd to move to the directory that you downloaded the file to, then install the program.

So I’ve got it.. but how do I use it ?

To start with use ‘sudo tcpdump’ to output to the terminal, this gives the standard output of TCPdump, using the default arguments since you have not passed TCPdump any arguments yet. To the output will be printed to the terminal, until you stop the program using Ctrl-C.Don’t be scared … we will now begin the break this output down into understandable section (if you not scared then try ‘sudo tcpdump -v’ or ‘sudo tcpdump -vv‘  for a verbose output).

Changing network interface
When I’m using tcpdump, it listened on eth0 by default, but you can change this. Entering ‘sudo tcpdump -D’ will return a list of network interfaces that you can choose between. For me this returns:
        1.eth0
        2.wlan0
        3.usbmon1 (USB bus number 1)
        4.usbmon2 (USB bus number 2)
        5.any (Pseudo-device that captures on all interfaces)
        6.lo
The first item in the list is the Ethernet port, the 2nd is the Wireless card and the 3,4,5th is self explanatory. The 6th is the loopback virtual interface (more information here on wikipedia)You can then change the interface, by calling TCPdump using ‘sudo tcpdump -i ‘. For most of my work, I use the local Wi-Fi so can change the interface from Ethernet to wireless using ‘sudo tcpdump -i wlan0’.

Creating a file to store the tcpdump arguments
So far, we have only added one argument when we call TCPdump, but there will be a lot more to come. We can save these arguments, which filter the output of TCPdump to a file and then use the file when we call TCPdump.Create the file using your favourite text editors (mine in gedit at the moment) and then pass the file to TCP dump using ‘sudo tcpdump -F ‘. The file dose not need any special file extension or layout.

You may need to change the file permissions so that TCPdump can extract the file. You can view the file premissions using ‘ls -l ‘. The ‘ls’ part is command line tool to list the files in a directory and the argument -l means use long format so the file permissions will be included. The file 10 characters are the file permissions, they are decoded as follows:
  1. ‘d’ means this is a directory and ‘-‘ means this is a file
  2. ‘r’ means that the file owner can read the file, ‘-‘ means they can’t
  3. ‘w’ means that the file owner can write to the file,  ‘-‘ means they can’t
  4. ‘x’  means that the file owner can execute the file,  ‘-‘ means they can’t. If this is a directory then ‘x’ means that the owner can list the files in directory, ‘-‘ means they can’t.
  5. ‘r’ means that the file group can read the file, ‘-‘ means they can’t
  6. ‘w’ means that the file group can write to the file,  ‘-‘ means they can’t
  7. ‘x’  means that the file group can execute the file,  ‘-‘ means they can’t. If this is a directory then ‘x’ means that the owner can list the files in directory, ‘-‘ means they can’t.
  8. ‘r’ means that everyone else can read the file, ‘-‘ means they can’t
  9. ‘w’ means that everyone else can write to the file,  ‘-‘ means they can’t
  10. ‘x’  means that everyone else can execute the file,  ‘-‘ means they can’t. If this is a directory then ‘x’ means that the owner can list the files in directory, ‘-‘ means they can’t.
For me, the default file permissions are ‘-rw-rw-r–‘. This therefore means that the file owner doesn’t have permission to execute the file. This can be changed using ‘chmod u+x ‘. The command line tool is chmod, this is used for changing file permissions, ‘u’ means we are considering the users permissions, ‘+’ means we are granting a permission and ‘x’ means we can considering the execute permission. Now, if I redo ‘ls -l ‘ the new result is -rwxrw-r–.
  
Sending the output to file
You can send the output of tcpdump to a file instead of printing it to the terminal using ‘sudo tcpdump -w ‘. This output is not saved as a plain text so you can’t just read it in a text editor instead you need to use a special program to read the file and extract information from it. You can do this using tcpdump by ‘tcpdump -r ‘. Alternatively you can open it using wireshark, launch wireshark and using File>Open.
 
Filtering information by protocol
To filter packets by protocol, add the name of protocol to the arguments of tcpdump so use something like ‘sudo tcpdump ‘. The main protocols that I’m likely are use are IP, ARP, TCP and UDP but others are available, see the man pages for a full list
Filtering information by direction to different hosts
To filter packets by a direction and host, add the direction and then the host name. Possible direction options are src, dst, src or dst, src and dst. You specify the host using its IP address. You can use the logical operators not, or and and. For example, you can look up the local IP address of your machine using ‘ifconfig ‘. If you don’t specify the name of the interface, then all interfaces will be listed. Now if you only want to view incoming traffic you can use ‘sudo udpdump dst ‘.
Change the level of detail 
Compared to the level of detail provided by the standard query, The detail can be reduced using ‘-q’ for quiet output or increased using ‘-v’ for verbose, ‘-vv’ for more verbose and ‘-vvv’ for even more verbose.
The opinion ‘-t’ means do not print timestamp on each line and the option ‘-a’ allows you to display each packet in ASCII or ‘-x’ to display in hex or ‘-X’ to display in hex and ASCII
View IP address instead of domain names
You can stop the translation of IP address to domain names using the ‘-n’ opinion and you can stop the translation of port number too using the ‘-nn’ opinion
Sources of Info
Wikipedia article, which contains very little information
Official public repo, including the man pages and FAQ
Useful online tutorial at openmaniak, this site also has good tutorials on networking tools that I’ve previously covered including wireshark, OpenVPN, Iperf and ping.