<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>BjOG - Bjou's Blog, that is! &#187; Tutorials</title>
	<atom:link href="http://www.bjou.de/blog/category/geek-talk/tutorials/feed/" rel="self" type="application/rss+xml" />
	<link>http://www.bjou.de/blog</link>
	<description>Whose Blog? Bjou's Blog!</description>
	<lastBuildDate>Sat, 17 Jul 2010 16:12:41 +0000</lastBuildDate>
	<generator>http://wordpress.org/?v=2.9.2</generator>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
			<item>
		<title>Howto: Copy/Tee/Clone network traffic using iptables</title>
		<link>http://www.bjou.de/blog/2008/05/howto-copyteeclone-network-traffic-using-iptables/</link>
		<comments>http://www.bjou.de/blog/2008/05/howto-copyteeclone-network-traffic-using-iptables/#comments</comments>
		<pubDate>Wed, 14 May 2008 22:51:22 +0000</pubDate>
		<dc:creator>bjou</dc:creator>
				<category><![CDATA[Geek Talk]]></category>
		<category><![CDATA[Tutorials]]></category>

		<guid isPermaLink="false">http://www.bjou.de/blog/?p=228</guid>
		<description><![CDATA[Having to work with Netflow data for my Diploma Thesis I invested quite some time into the following challenge:
Our Routers export Cisco Netflow Data to HOST A, where we do accounting. I want to use HOST B for several Netflow-related tests. The Routers only support one target for their netflow export (as mentioned, this target [...]]]></description>
			<content:encoded><![CDATA[<p>Having to work with Netflow data for my Diploma Thesis I invested quite some time into the following challenge:<br />
Our Routers export Cisco Netflow Data to <em>HOST A</em>, where we do accounting. I want to use <em>HOST B</em> for several Netflow-related tests. The Routers <em>only support one target</em> for their netflow export (as mentioned, this target is HOST A).</p>
<p><strong>Problem: How is it possible to clone the incoming stream of packets at HOST A and forward one copy into HOST A&#8217;s userspace (for accounting applications) and the other copy to HOST B&#8217;s userspace (for testing purposes)?</strong><br />
The specific challenge is that I do not want a simple FORWARD to HOST B, but a FORWARD of a copy, so that I can work with the data on both machines. This leads to the next problem: Packets arriving at HOST B have the Destination IP address of HOST A in their IP header. We need to rewrite this IP at HOST B so that userspace applications are able to process these packets (which they are not, if the packets are not destined to HOST B&#8217;s address).</p>
<div align="center">
<a href='http://www.bjou.de/blog/wp-content/uploads/iptables_tee.jpg'><img src="http://www.bjou.de/blog/wp-content/uploads/iptables_tee.jpg" alt="" title="iptables_tee"  class="aligncenter size-full wp-image-229" /></a></div>
<p><span id="more-228"></span><br />
<strong>Note in advance</strong>: Despite all efforts this tutorial only works for connectionless udp traffic. A successful 3-way-handshake on HOST A prevents HOST B (despite IP-address rewriting) from accepting the packets in userspace. It just does not work, I appreciate any comments on that. Remember that <em>tee</em> is normally used to clone traffic to another host for passive sniffing and traffic analysis. Note as well, that even if you might want to keep this approach centralized and rewrite the packet&#8217;s IP addresses already at HOST A in the <em>POSTROUTING </em>chain, this will not work: Teed packets do not yet show up anywhere within the iptables structures to avoid interfering with the original packet&#8217;s table traversal. This is subject to change, though. Thanks to Jan Engelhardt for this information.</p>
<p>So here is how we achieve this goal (tested on Debian Etch stable):<br />
<strong>History</strong>: There used to be a <em>tee </em>option for an experimental <em>ROUTE target</em>,  patchable into iptables with <a href="http://www.netfilter.org/projects/patch-o-matic/pom-extra.html">patch-o-matic</a> (pom). This will not work on recent kernels and is deprecated!</p>
<p>This is what we will do on <strong>HOST A</strong>: Get xtables-addons from <a href="http://dev.computergmbh.de/">http://dev.computergmbh.de</a>:<br />
<code>wget http://dev.computergmbh.de/files/xtables/xtables-combined-1.5.4.1.tar.bz2</code><br />
This includes a current snapshot of iptables.</p>
<blockquote><p>Xtables-addons is the proclaimed successor to patch-o-matic(-ng). It<br />
contains extensions that were not accepted in the main Xtables<br />
package.<br />
Xtables-addons is different from patch-o-matic in that you do not have<br />
to patch or recompile either kernel or Xtables(iptables).</p></blockquote>
<p>Untar, configure, make and make install. Should you run into problems of the kind<br />
<code>warning: #warning You need either CONFIG_NF_CONNTRACK or CONFIG_IP_NF_CONNTRACK</code><br />
 change into your kernel source directory and adapt your kernel. Therefore, look for the Networking option, find the Netfilter (formerly know as ipchains) framework entry and enable the appropriate options. I also ran into problems saying<br />
<code>warning: #warning You have CONFIG_IP_NF_CONNTRACK enabled, but CONFIG_IP_NF_CONNTRACK_MARK or CONFIG_IP_NF_CONNTRACK_SECMARK are not (please enable)</code><br />
so be sure to enable these options as well in the <em>IP: Netfilter Configuration</em> section.<br />
Save your config and build a new kernel. However, this is not topic of this tutorial.<br />
Should there be other errors because of a special addon, deactivate it in the xtables-addons directory using the <em>mconfig</em> file. Be sure not to deactivate the <em>TEE</em> target, as this is the one we need. The installation success of the xtables-addons may largely depend on the kernel that is being used. If it just won&#8217;t work for you with your existing kernel, try another one. I had successful setups on 2.6.23.16 and 2.6.18.6</p>
<p>After successful installation fire the command<br />
<code>iptables -t mangle -A PREROUTING -p udp --dport 9996 -j TEE --gateway &lt;IP of HOST B&gt;</code>.<br />
This command will clone all incoming udp-packets to port 9996 in kernelspace and copy them to HOST B, where we will rewrite the IP addresses. Confirm by typing<br />
<code>iptables -t mangle -L</code><br />
This will list your rules in the <em>mangle</em> table.<br />
Should there be an error about an unknown table/target/chain, then xtables-addons did not build/install successful, probably because of some missing kernel options.</p>
<p><strong>On HOST B</strong>: You do not need xtables-addons here, but only some standard iptables version, as you only need the default DNAT target (Your kernel needs to support it however. Therefore, make sure to have the <em>IPv4 connection tracking support (required for NAT)</em> option enabled in the <em>IP: Netfilter Configuration</em> section of your netfilter kernel category).<br />
<code>iptables -t nat -A PREROUTING -p udp -d &lt;IP of HOST A&gt; --dport 9996 -j DNAT --to-destination &lt;IP of HOST B&gt;:&lt;Port&gt;</code></p>
<p>That should be it. Now test your setup. You will need three hosts: HOST A and B and another HOST C where you will generate (UDP-) packets. Get a packet generator (I used <a href="http://ip-packet.sourceforge.net/">IP-Packet</a>) and download it to HOST C. Read its documentation, create a config file and fire up your packets to <em>HOST A port 9996</em>. But first, make sure you have a listening process on both, HOST A and B running and waiting for your packets on that specific port. The easiest way will be to use netcat in udp-mode:<br />
<code>nc -ulp9996</code> on HOST A and on HOST B respective with the port used there. Fire your packets and both netcat instances should receive the UDP payload data. If only HOST A gets them, your <em>tee </em>or <em>DNAT </em>is not working. Debug yourself <img src='http://www.bjou.de/blog/wp-includes/images/smilies/icon_sad.gif' alt=':(' class='wp-smiley' />  That&#8217;s what I need to do now, as well, because teeing seems to work perfectly fine from one host, but not from the other&#8230; Same settings, though, this is just not fair :&#8217;(</p>
]]></content:encoded>
			<wfw:commentRss>http://www.bjou.de/blog/2008/05/howto-copyteeclone-network-traffic-using-iptables/feed/</wfw:commentRss>
		<slash:comments>6</slash:comments>
		</item>
		<item>
		<title>Intrusion Detection with Heterogenous Sensors</title>
		<link>http://www.bjou.de/blog/2007/08/intrusion-detection-with-heterogenous-sensors/</link>
		<comments>http://www.bjou.de/blog/2007/08/intrusion-detection-with-heterogenous-sensors/#comments</comments>
		<pubDate>Fri, 24 Aug 2007 17:48:11 +0000</pubDate>
		<dc:creator>bjou</dc:creator>
				<category><![CDATA[Geek Talk]]></category>
		<category><![CDATA[Karlsruhe/Uni]]></category>
		<category><![CDATA[Tutorials]]></category>

		<guid isPermaLink="false">http://www.bjou.de/blog/2007/08/intrusion-detection-with-heterogenous-sensors/</guid>
		<description><![CDATA[My study thesis is finally completed. It deals with different Intrusion Detection approaches consisting mainly of sniffers and honeypots and their implementation in the University&#8217;s campus network. Basically, I have deployed several heterogenous sensors in different subnets/VLANs and enabled all of them to report to one centralized console for further investigations and automated incident forwarding [...]]]></description>
			<content:encoded><![CDATA[<p>My study thesis is finally completed. It deals with different Intrusion Detection approaches consisting mainly of sniffers and honeypots and their implementation in the University&#8217;s campus network. Basically, I have deployed several heterogenous sensors in different subnets/VLANs and enabled all of them to report to one centralized console for further investigations and automated incident forwarding to the appropriate persons in charge. This is the abstract:</p>
<blockquote><p>Computer systems and -networks connected to the Internet are exposed to a large array of malicious activities. Computers throughout the world are continuously being scanned for vulnerabilities, exploited and finally compromised by humans or by autonomously spreading malicious software, called malware. To stem this thread, the University of Karlsruhe has deployed an Intrusion Detection System (IDS) consisting of the Intrusion Prevention module ”IntruShield” and some other quarantine automation modules. The purpose of this study thesis is to ensure deeper security by extending this IDS according to the ”Defense in Depth” strategy. Therefore, several sensors have been deployed into the core- and user network to extend the current setup to a distributed Intrusion Detection System. The heterogeneity of these sensors aid to cover different kinds of attacks: Sniffers for network traffic examination, honeypots for network-host specific operations and host-based IDSs (HIDS) for hostspecific activities. As this heterogeneity leads to a large amount of different data to be analyzed, it has been decided to implement a hybrid Intrusion Detection framework, which enables all the different security applications, i.e. sensors, to report to a centralized console which performs automatic aggregation of the distributed data and correlation between the various events, presented to the security analyst through a web interface.<br />
This work gives an introduction to the basic principles, approaches and securityrelated software technologies used throughout this study thesis. Moreover, it describes the current security concept of the University of Karlsruhe as well its proposed enhancements. In detail, this thesis evinces the campus-wide implementation of the previously mentioned distributed Intrusion Detection architecture and concludes with its evaluation and a future outlook. The thesis shows that the commissioning of this approach results not only in a better automated, but also in a more structured, more unified and a more accelerated security process.</p></blockquote>
<p><strong>Link:</strong> <a href="http://www.bjou.de/blog/wp-content/uploads/intrusion_detection_with_heterogenous_sensors.pdf">Intrusion Detection with Heterogenous Sensors</a>, 86 pages, 3.5 MB.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.bjou.de/blog/2007/08/intrusion-detection-with-heterogenous-sensors/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Computerbefehle durch Zupfeifen ausführen</title>
		<link>http://www.bjou.de/blog/2007/01/computerbefehle-durch-zupfeifen-ausfuhren/</link>
		<comments>http://www.bjou.de/blog/2007/01/computerbefehle-durch-zupfeifen-ausfuhren/#comments</comments>
		<pubDate>Thu, 11 Jan 2007 12:02:58 +0000</pubDate>
		<dc:creator>bjou</dc:creator>
				<category><![CDATA[Geek Talk]]></category>
		<category><![CDATA[Tutorials]]></category>

		<guid isPermaLink="false">http://www.bjou.de/blog/2007/01/computerbefehle-durch-zupfeifen-ausfuhren/</guid>
		<description><![CDATA[Wer schon immer mal seinen Computer durch simples Pfeifen steuern wollte, wie es in diesem Youtube Video gemacht wird, sollte dieses kleine Tutorial lesen. Es sollte mit Debian und Ubuntu funktionieren. Zunächst installiert man die Pakete libasound2-dev und freeglut3-dev. Das Paket xwit ist für fortgeschrittene Fenster-Steuerung sehr interessant, wird aber für die Basis-Funktionen nicht benötigt. [...]]]></description>
			<content:encoded><![CDATA[<p>Wer schon immer mal seinen Computer durch simples Pfeifen steuern wollte, wie es in <a href="http://youtube.com/watch?v=x9jHReFtYgE">diesem Youtube Video</a> gemacht wird, sollte dieses kleine Tutorial lesen. Es sollte mit Debian und Ubuntu funktionieren. <span id="more-150"></span>Zunächst installiert man die Pakete <em>libasound2-dev</em> und <em>freeglut3-dev</em>. Das Paket <em>xwit</em> ist für fortgeschrittene Fenster-Steuerung sehr interessant, wird aber für die Basis-Funktionen nicht benötigt. Nun besorgt man sich <a href="http://www.cs.princeton.edu/sound/software/sndpeek/">hier</a> die sndpeek Programm-Quellen, editiert die Datei <em>src/sndpeek/sndpeek.cpp</em>, so dass aus dem Codeschnipsel (etwa Zeile 1108)</p>
<p><code><br />
    fprintf( stdout, "%.2f %.2f %.2f %.2f %.2f %.2f %.2f %.2f %.2f %.2f %.2f %.2f %.2f ",<br />
         mfcc(0), mfcc(1), mfcc(2), mfcc(3), mfcc(4), mfcc(5), mfcc(6),<br />
         mfcc(7), mfcc(8), mfcc(9), mfcc(10), mfcc(11), mfcc(12) );<br />
    fprintf( stdout, "\n" );<br />
</code></p>
<p>dieser hier wird</p>
<p><code><br />
    fprintf( stdout, "%.2f %.2f %.2f %.2f %.2f %.2f %.2f %.2f %.2f %.2f %.2f %.2f %.2f ",<br />
         mfcc(0), mfcc(1), mfcc(2), mfcc(3), mfcc(4), mfcc(5), mfcc(6),<br />
         mfcc(7), mfcc(8), mfcc(9), mfcc(10), mfcc(11), mfcc(12) );<br />
    fprintf( stdout, "\n" );<br />
    fflush(stdout);<br />
</code></p>
<p>Ein <em>make linux-alsa</em> im Verzeichnis <em>/src/sndpeek</em> und ein anschließendes <em>make install</em> installiert die gepatchte Software. Es sind auch andere Optionen möglich, bspw: make linux-jack, make linux-oss, make osx, or make win32, je nach OS und Soundsystem. Jetzt kann man sich das Perl-Script <a href="http://www-128.ibm.com/developerworks/views/download.jsp?contentid=186736&#038;filename=os-whistle.zip&#038;method=http&#038;locale=worldwide">os-whistle</a> besorgen, das entpackt wird und nun zum Konfigurieren wie folgt aufgerufen wird:</p>
<p><code>sndpeek &#45;&#45;print &#45;&#45;nodisplay | perl cmdWhistle.pl -c</code> Dies lässt sndpeek am Mikrophon lauschen und den Output an das Perl-Script übergeben. Lässt man &#45;&#45;nodisplay weg, hat man dazu noch eine nette 3D-Ausgabe. Man kann nun beginnen, ins Mikrofon zu pfeifen und den Output zu beobachten. Ein Timeout erfolgt nach 4 sec Stille. Die Ausgabe sieht ähnlich der folgenden aus:</p>
<p><code>25.00 25.00 _#_ 0 500000 _#_ &lt;command here&gt; _#_ &lt;comment here&gt;.</code> Jetzt kopiert man, wie vorgeschlagen, diesen Output in die Datei <em>.toneFile</em> ins Home Verzeichnis, wobei man <em>&lt;command here&gt;</em> durch ein beliebiges Kommando ersetzt, bspw. <em>mozilla-thunderbird</em>. Ein erneuter Start des Programms mit<br />
<code>sndpeek &#45;&#45;print &#45;&#45;nodisplay | perl cmdWhistle.pl</code> (optional mit -v für verbose Output) startet den Daemon. Pfeift man nun nochmal die gleiche Tonsequenz, startet sich das eMailprogramm und checkt die Mails&#8230; nur durch Pfeifen gesteuert, weeeeeeeh <img src='http://www.bjou.de/blog/wp-includes/images/smilies/icon_wink.gif' alt=';)' class='wp-smiley' /> </p>
<p>Mehr Infos und Details gibt es bei <a href="http://www-128.ibm.com/developerworks/library/os-whistle/index.html?ca=dgr-lnxw97whistlework">IBM</a>.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.bjou.de/blog/2007/01/computerbefehle-durch-zupfeifen-ausfuhren/feed/</wfw:commentRss>
		<slash:comments>1</slash:comments>
		</item>
		<item>
		<title>Advanced Packet Capturing Howto: PF_RING, NAPI and extended libpcap on Debian Sarge</title>
		<link>http://www.bjou.de/blog/2006/12/advanced-packet-capturing-howto-pf_ring-napi-and-extended-libpcap-on-debian-sarge/</link>
		<comments>http://www.bjou.de/blog/2006/12/advanced-packet-capturing-howto-pf_ring-napi-and-extended-libpcap-on-debian-sarge/#comments</comments>
		<pubDate>Wed, 20 Dec 2006 15:51:12 +0000</pubDate>
		<dc:creator>bjou</dc:creator>
				<category><![CDATA[Geek Talk]]></category>
		<category><![CDATA[IT-Security]]></category>
		<category><![CDATA[Tutorials]]></category>

		<guid isPermaLink="false">http://www.bjou.de/blog/2006/12/advanced-packet-capturing-howto-pf_ring-napi-and-extended-libpcap-on-debian-sarge/</guid>
		<description><![CDATA[Working on my student research project, I have to monitor a quite large network. Therefore, I have configured one of the main switches to mirror the traffic to my hi-end sniffing machine. Trying to capture the traffic with software depending on libpcap, I encountered massive packet loss almost immediately using libpcap-0.8 on Debian (which is [...]]]></description>
			<content:encoded><![CDATA[<p>Working on my student research project, I have to monitor a quite large network. Therefore, I have configured one of the main switches to mirror the traffic to my hi-end sniffing machine. Trying to capture the traffic with software depending on libpcap, I encountered massive packet loss almost immediately using libpcap-0.8 on Debian (which is version 0.9.5-1). Now I have commited a whole day in trying to decrease the amount of dropped packets. There are some promising solutions, namely <a href="http://public.lanl.gov/cpw/">mmap&#8217;ed libpcap</a>, NAPI (polling-enabled Network Driver) and <a href="http://www.ntop.org/PF_RING.html">PF_RING</a>, the latter being the most promising, after having read &#8220;<a href="http://luca.ntop.org/Ring.pdf">Improving Passive Packet Capture: Beyond Device Polling</a>&#8220;.  Now theory sounds great and is one thing, but getting it to run without any useful documentation almost killed me. Now here is how I finally did it&#8230;<br />
<span id="more-144"></span></p>
<p><!--adsense#linkblockbreit--><br />
First of all, I wanna thank <a href="http://synfulpacket.blogspot.com/2006/11/pfring-and-snort.html">Richard</a> for his in-depth tutorial, which I unfortunately found in a quite late state of my efforts. Anyhow, his tutorial is based on a RedHat Enterprise 4 System, but most of it is usable on other systems, too. It did not fit completely on Debian, though. I copied part of his text for simplicity. So let&#8217;s get it on:</p>
<ul>
<li>Remove libpcap* via apt-get or aptitude and all other software that depends on it (you gotta rebuild it later with the new pfring-enabled libpcap)</li>
<li>Rebuild your kernel:<br />
<code>cd /usr/src<br />
export CVSROOT=:pserver:anonymous@cvs.ntop.org:/export/home/ntop<br />
mkdir pf_ring &amp;&amp; cd pf_ring<br />
cvs login</code><br />
which should produce the following output:</p>
<p><code>Logging in to :pserver:anonymous@cvs.ntop.org:2401/export/home/ntop<br />
CVS password:</code></p>
<p>At the prompt, type &#8220;ntop&#8221; (no quotes), and hit Enter. (Note: ntop will not appear on the screen.) Next type the following:</p>
<p><code>cvs checkout PF_RING</code> which will download the needed source.</li>
<li>Now etner the directory and edit the file mkpatch.sh. Therefore, check which kernel you currently have installed or which kernel you want to have after the update. Check by <code>uname -a</code> which should give you something like <em>Linux rz-sniff 2.6.17.11 #1 SMP Wed Dec 20 13:06:04 CET 2006 i686 GNU/Linux</em>. Open mkpatch.sh, scroll to the first variable declarations and change into your wishes:<br />
<code>VERSION=${VERSION:-2}<br />
PATCHLEVEL=${PATCHLEVEL:-6}<br />
SUBLEVEL=${SUBLEVEL:-17.11}<br />
</code>and run the script via <em>sh ./mkpatch.sh</em>, which should give you some output and stop with the location of your patchfile.</li>
<li>Apply the patchfile:<br />
<code>cd /usr/src<br />
zcat /usr/src/pf_ring/PF_RING/workspace/linux-2.6.*patch.gz | patch --dry-run -p0</code> If it runs without any error, run again without the <em>&#8211;dry-run</em> option</li>
<li>Now you gotta build your custom kernel. This may vary on different systems, a 2.6 kernel under Debain Sarge can be build as followed:<br />
<code>cd /usr/src/linux<br />
apt-get install kernel-package libncurses5-dev fakeroot wget bzip2 build-essential<br />
make menuconfig</code><br />
It is normally a good idea to take the configuration of your existing (working!) kernel 2.6 as a starting point for the configuration of your new kernel. Usually the current kernel configuration is saved in a file under /boot, e.g. /boot/config-2.6.17.11. Load it using the arrow keys and selection the option at the bottom.<br />
Now for Intel Network Cards, select <em>Device Driver &#8211; Network Service Support &#8211; Ethernet (1000Mbit)</em> and enable <strong>NAPI-Support</strong> by selecting &#8216;y&#8217; on <em>Use Rx Polling</em>. Go back (3x ESC) and select <em>Networking </em>and <em>Networking Options</em>. Make sure that <em><strong>PF_RING sockets</strong></em> are enabled. Leave and save the settings. Now build the kernel and install it:<br />
<code>make-kpkg clean<br />
fakeroot make-kpkg --initrd --revision=pfring.1.0 kernel_image<br />
cd ../<br />
dpkg -i linux-image-2.6.17.11_pfring.1.0_i386.deb<br />
</code><br />
This will install your new kernel (including a ramdisk) and also update grub. You can now reboot your system, and you should then have a new kernel. You can check that by running <em>uname -r</em>.</li>
<li>Compile libpfring, and test that it works.<br />
First off, start with doing the following:</p>
<p><code>cp /usr/src/linux/include/linux/ring.h /usr/include/linux</code></p>
<p>which will add the neccessary header file, ring.h, to the standard include directory.</p>
<p>Next, run the following:</p>
<p><code>cd /usr/src/pf_ring/PF_RING/userland/libpfring<br />
make</code></p>
<p>which should give you a short output.<br />
Now, we are going to generate a .so from these files, by running the following:</p>
<p><code>gcc -shared -Wl,-soname -Wl,libpfring.so.0.9.4 -o libpfring.so.0.9.4 *.o -lc</code></p>
<p>We copy the files we just created to a common system directory:</p>
<p><code>cp libpfring.a libpfring.so.0.9.4 /usr/local/lib<br />
cp pfring.h /usr/local/include<br />
ln -s /usr/local/lib/libpfring.so.0.9.4 /usr/local/lib/libpfring.so<br />
</code></p>
<p>Next, we need to add /usr/local/lib to the list of directories the dynamic loader will search:<br />
<code><br />
echo "/usr/local/lib" &gt;&gt; /etc/ld.so.conf<br />
ldconfig</code></p>
<p>To check that the dynamic loader sees the libraries, run the following:</p>
<p><code>ldconfig -v |grep pfring</code></p>
<p>which should produce the following output:</p>
<p><em>libpfring.so.0.9.4 -&gt; libpfring.so.0.9.4</em></p>
<p>You can test the installation with</p>
<p><code>./pfcount -v -i eth<em>x</em></code> with x being your interface number. Be carefull, the thing can hang on a very busy interface (e.g. &gt; 100Mbit)</p>
<p>Now, run the following:</p>
<p><code>dmesg</code></p>
<p>and the last few lines of output should look similar to the following:</p>
<p><code>RING: succesfully allocated 128 KB [tot_mem=26509372][order=5]<br />
RING: allocated 80 slots [slot_len=1618][tot_mem=131072]<br />
device eth1 entered promiscuous mode</code></li>
<li>Next step is to build libpcap to use the PF_RING interface to the kernel.<br />
First, run the following:</p>
<p><code>cd /usr/src/pf_ring/PF_RING/userland/<br />
ls |grep pcap</code></p>
<p>The output should be something similar to:</p>
<p><em>libpcap-0.9.4-ring</em></p>
<p>which indicates that this version of PF_RING was built to work with a patched version of libpcap-0.9.4, so we need to download that version.</p>
<p>The simplest way to download that version is to run the following:<br />
<code><br />
wget http://www.tcpdump.org/release/libpcap-0.9.4.tar.gz</code> Now do:</p>
<p><code>tar -zxvf libpcap-0.9.4.tar.gz<br />
cd libpcap-0.9.4<br />
mv pcap-int.h pcap-int.h.orig<br />
mv pcap-linux.c pcap-linux.c.orig<br />
cp ../libpcap-0.9.4-ring/pcap* .<br />
./configure CPPFLAGS="-I/usr/local/include" LDFLAGS="-L/usr/local/lib" CFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64"<br />
</code></p>
<p>If the ./configure command completes without any errors, run the following:</p>
<p><code>make &amp;&amp; gcc -shared -Wl,-soname -Wl,libpcap.so.`cat VERSION` -o libpcap.so.`cat VERSION` *.o -lc</code></p>
<p>Now, run the following:</p>
<p><code>make install &amp;&amp; cp libpcap.so.0.9.4 /usr/local/lib</code></p>
<p>Next, make sure the dynamic loader sees this new library:</p>
<p><code>ldconfig -v |grep pcap</code></p>
<p>the output should look similar to:</p>
<p><em>libpcap.so.0.9.4 -&gt; libpcap.so.0.9.4</em></li>
<li>Compile your software using the new libpcap and pfring, e.g. ntop. If you are getting ntop from CVS also (like PF_RING), change into the ntop directory and compile it, using the following commands. These can be used with other software, too (i.e. the compile options)<code>./autogen.sh --noconfig<br />
./configure CPPFLAGS="-I/usr/local/include" LDFLAGS="-L/usr/local/lib -lpfring -lpcap"<br />
make &amp;&amp; make install</code></p>
<p>Now check if your setup was successful:</p>
<p><code>ldd /usr/local/bin/ntop</code></p>
<p>should produce output similar to:</p>
<p><code> linux-gate.so.1 =&gt;  (0xffffe000)<br />
libpfring.so.0.9.4 =&gt; /usr/local/lib/libpfring.so.0.9.4 (0xb7f8c000)<br />
libntopreport-3.2.4.so =&gt; /usr/local/lib/libntopreport-3.2.4.so (0xb7ee6000)<br />
libntop-3.2.4.so =&gt; /usr/local/lib/libntop-3.2.4.so (0xb78f9000)<br />
libpthread.so.0 =&gt; /lib/tls/i686/cmov/libpthread.so.0 (0xb78e7000)<br />
libdl.so.2 =&gt; /lib/tls/i686/cmov/libdl.so.2 (0xb78e3000)<br />
libcrypt.so.1 =&gt; /lib/tls/i686/cmov/libcrypt.so.1 (0xb78b5000)<br />
libc.so.6 =&gt; /lib/tls/i686/cmov/libc.so.6 (0xb7784000)<br />
libresolv.so.2 =&gt; /lib/tls/i686/cmov/libresolv.so.2 (0xb7771000)<br />
libnsl.so.1 =&gt; /lib/tls/i686/cmov/libnsl.so.1 (0xb775a000)<br />
libpcap.so.0.9.4 =&gt; /usr/local/lib/libpcap.so.0.9.4 (0xb772f000)<br />
libgdbm.so.3 =&gt; /usr/lib/libgdbm.so.3 (0xb7729000)<br />
libgd.so.1 =&gt; /usr/lib/libgd.so.1 (0xb76f7000)<br />
libpng12.so.0 =&gt; /usr/lib/libpng12.so.0 (0xb76d4000)<br />
libz.so.1 =&gt; /usr/lib/libz.so.1 (0xb76c0000)<br />
librrd_th.so.2 =&gt; /usr/lib/librrd_th.so.2 (0xb7679000)<br />
/lib/ld-linux.so.2 (0xb7f99000)<br />
libm.so.6 =&gt; /lib/tls/i686/cmov/libm.so.6 (0xb7654000)<br />
libjpeg.so.62 =&gt; /usr/lib/libjpeg.so.62 (0xb7634000)<br />
libfreetype.so.6 =&gt; /usr/lib/libfreetype.so.6 (0xb75ca000)<br />
libart_lgpl_2.so.2 =&gt; /usr/lib/libart_lgpl_2.so.2 (0xb75b4000)</code></p>
<p>where the line containing libpfring and libpcap are of particular importance.</p>
<p>At this point, you have a version of ntop that will use the PF_RING ring module in the kernel. Congrats.</p>
<p><!--adsense#anzeigenblocknurtext--></li>
<li>Some Statistics:<code>cat /proc/net/pf_ring/info</code><br />
will give you some output like this:</p>
<p><code>Version       : 3.2.1<br />
Bucket length : 128 bytes<br />
Ring slots    : 4096<br />
Sample rate   : 1 [1=no sampling]<br />
Capture TX    : No [RX only]<br />
Total rings   : 0<br />
</code></p>
<p>Now there is one drawback: libpcap will not report accurate drop statistics when linked with pfring, so the 0 dropped packets, that ntop reports, are definately wrong.</p>
<p><code>cat /proc/net/pf_ring/20</code> (where 20 is some random number) shows the actual stats:</p>
<p><code>Bound Device  : eth0<br />
Version       : 6<br />
Sampling Rate : 0<br />
Cluster Id    : 0<br />
Tot Slots     : 7181<br />
Slot Len      : 146<br />
Data Len      : 128<br />
Tot Memory    : 1048576<br />
Tot Packets   : 62206634<br />
Tot Pkt Lost  : 14292126<br />
Tot Insert    : 47914508<br />
Tot Read      : 47914434<br />
</code></p>
<p>Do the math (and wisely distinguish between capture- and drop-rate): 14292126 out of 62206634 is a <em>drop</em>-rate of about 23%. Sounds high? Well, it isn&#8217;t, regarding the fact, that my network has peaks up to and over 600Mbps and a current average of about 150Mbps after a 30min runtime. Viewing table 3 in the PDF mentioned in the introduction, you can see, that the performance highly depends on the packet size.<br />
Now 55% of my traffic has sizes between 60 and 256 bytes (i.e. small packets), for which the table promises a <em>capture</em>-rate of 75%. Another 35% of my traffic is above 1025 bytes, where a Linux 2.6 with NAPI + PF_RING and extended libpcap performs best (93% <em>capture</em>-rate). It is quite poor in between (47% <em>capture</em>-rate), but still the best of its competitors. Anyhow, regarding the fact, that most of my packets are big or small, it was a wise choice to use this specific polling strategy, and a 23% <em>drop</em>-rate is about the result I have expected. Tests with standard kernel and libpcap had <em>drop</em>-rates of up to 40%.<br />
In my study thesis &#8220;<a href="http://www.bjou.de/blog/wp-content/uploads/intrusion_detection_with_heterogenous_sensors.pdf">Intrusion Detection with heterogenous Sensors</a>&#8221; you can find detailed long-term statistics in the Evaluation Chapter (Chapter 5), &#8220;<em>Sniffing Performance</em>&#8221; featuring nice graphs that illustrate the work of the ringbuffer.</p>
<p>/var/log/messages should have some stats, too, upon every application using pf_ring<br />
<code><br />
Welcome to PF_RING 3.2.1<br />
(C) 2004-06 L.Deri<br />
NET: Registered protocol family 27<br />
PF_RING: bucket length    128 bytes<br />
PF_RING: ring slots       4096<br />
PF_RING: sample rate      1 [1=no sampling]<br />
PF_RING: capture TX       No [RX only]<br />
PF_RING: transparent mode Yes<br />
PF_RING initialized correctly.<br />
PF_RING: registered /proc/net/pf_ring/<br />
RING: succesfully allocated 1024 KB [tot_mem=598076][order=8]<br />
RING: allocated 7181 slots [slot_len=146][tot_mem=1048576]<br />
</code></p>
<p>Now it is possible to manually remove and insert the kernelmodule into the kernel.<br />
<em>lsmod</em> shows if ring.ko is loaded and used, <em>rmmod</em> removes it and it can be loaded with configurable parameters via</p>
<p><code><br />
insmod /lib/modules/2.6.17.11/kernel/net/ring/ring.ko bucket_len=64 num_slots=4096 sample_rate=1 transparent_mode=0</code></p>
<p><strong>bucket_len</strong>: Specifies the maximum packet length captured by PF_RING. This is equivalent to snaplen of libpcap. If you are doing something like ntop where you only want to look at the packet headers, then a bucket_len of 64 works.  But if you want to inspect the entire packet, then you will have to make sure that the bucket_len is at least as big as the MTU<br />
of the network.<br />
<strong>num_slots</strong>: Number of slots in the ring. The bigger the better is the performance the more memory you use. 4096 should be fine, you can tweak it though.<br />
<strong>sample_rate</strong>: 1 means regard every packet (no sampling), 2 means every second and so on&#8230;<br />
<strong>transparent_mode</strong>:  By default a packet that is handled by at least a ring is not forwarded to the upper Linux layers. This will result in faster capture speeds but will prevent legacy applications (not recompiled with the new libpcap-ring) from operating. If you set it to 1 it reverts the ring to the old behaviour (i.e. packets are forwarded to upper layers) but this will decrease the benefits of the ring as it will result in worse results.</p>
<p>Tweak it according to your purpsose.</p>
<p>To conclude: I don&#8217;t know if it makes any difference, but by checking your NIC via</p>
<p><code>ethtool -g eth0</code> you can see the internal ringbuffer settings. You can change it to a greater value using<br />
<code><br />
ethtool -G eth0 rx 4096</code></li>
</ul>
<p>&#8212;&#8212;&#8212;&#8212;</p>
<p style="text-align: center;"><a href="http://www.dealdoktor.de"><strong>Top-Schnäppchen &amp; Gutscheine: DealDoktor</strong></a> &#8211; Der einzige <strong>Schnäppchen-Blog mit Doktortitel</strong></p>
<p style="text-align: center;">Hier gibt es viele <strong>Gutscheine</strong>, <strong>Gratis-Artikel</strong> und täglich die besten <strong>Spar-Angebote</strong> aus dem Internet, z.B. Möglichkeiten, kostenlos ins Kino zu kommen und mehr&#8230;</p>
]]></content:encoded>
			<wfw:commentRss>http://www.bjou.de/blog/2006/12/advanced-packet-capturing-howto-pf_ring-napi-and-extended-libpcap-on-debian-sarge/feed/</wfw:commentRss>
		<slash:comments>8</slash:comments>
		</item>
		<item>
		<title>Howto: getmail, dovecot, postfix, spamassassin und procmail</title>
		<link>http://www.bjou.de/blog/2006/06/getmail-dovecot-postfix-spamassassin-und-procmail/</link>
		<comments>http://www.bjou.de/blog/2006/06/getmail-dovecot-postfix-spamassassin-und-procmail/#comments</comments>
		<pubDate>Thu, 22 Jun 2006 16:20:48 +0000</pubDate>
		<dc:creator>bjou</dc:creator>
				<category><![CDATA[Geek Talk]]></category>
		<category><![CDATA[IT-Security]]></category>
		<category><![CDATA[Mein Tach]]></category>
		<category><![CDATA[Tutorials]]></category>

		<guid isPermaLink="false">http://www.bjou.de/blog/?p=96</guid>
		<description><![CDATA[Dieses kleine Howto soll kurz erklären, wie die genannten Programme reibungslos Hand in Hand arbeiten und so einen IMAP-Dienst mit Spamfilterung und Mailverteilung realisieren können. Das Tutorial basiert auf dem Artikel &#8220;Mailhamster mit FreeBSD&#8221; und erweitert ihn um die Konfiguration von Spamassassin und Procmail. User, die noch mit sendmail arbeiten, sollten auf postfix umstellen. Ein [...]]]></description>
			<content:encoded><![CDATA[<p>Dieses kleine Howto soll kurz erklären, wie die genannten Programme reibungslos Hand in Hand arbeiten und so einen IMAP-Dienst mit Spamfilterung und Mailverteilung realisieren können. Das Tutorial basiert auf dem Artikel &#8220;<a href="http://wiki.bsdforen.de/index.php/Mailhamster_mit_FreeBSD">Mailhamster mit FreeBSD</a>&#8221; und erweitert ihn um die Konfiguration von Spamassassin und Procmail. User, die noch mit sendmail arbeiten, sollten auf postfix umstellen. Ein simples Howto dazu gibt es <a href="http://www.csua.berkeley.edu/~ranga/notes/freebsd_postfix.html">hier</a>.<br />
<span id="more-96"></span><br />
Das Mailhamster Tutorial sollte zunächst wie beschrieben nachvollzogen werden, lediglich mit der Konfiguration von procmail kann gewartet werden, da wir procmail an spamassassin anpassen müssen.</p>
<p>Nachdem also getmail und dovecot (wie im Tutorial beschrieben) zusammenarbeiten, machen wir uns an die Installation von Spamassassin:</p>
<p><code>cd /usr/ports/mail/p5-Mail-SpamAssassin &#038;&#038; make install clean</code></p>
<p>Unter <em>/usr/local/etc/mail/spamassassin</em> gibt es nun eine local.cf.sample, die man in local.cf umbenennt. Diese Datei steuert das Verhalten von spamassassin. Sie kann nun angepasst werden (wenn man weiß, was man tut) oder mit Hilfe eines selbsterklärenden Webinterfaces erstellt werden: <a href="http://www.yrex.com/spam/spamconfig.php">http://www.yrex.com/spam/spamconfig.php</a>. Es ist noch anzumerken, dass in dem genannten Verzeichnis die globale Config-File von Spamassassin liegt. Wird spamassassin mit Hilfe des -u Flags im Kontext eines bestimmten Users ausgeführt, so ist die Konfigurationsdatei für diesen User so anzulegen: ~/.spamassassin/user_prefs (mit gleichem Inhalt).</p>
<div align="center"<br />
<!--adsense-->
</div>
<p>Der Inhalt meiner Datei sieht so aus:<br />
<code># SpamAssassin config file for version 3.x<br />
# NOTE: NOT COMPATIBLE WITH VERSIONS 2.5 or 2.6<br />
# See http://www.yrex.com/spam/spamconfig25.php for earlier versions<br />
# Generated by http://www.yrex.com/spam/spamconfig.php (version 1.50)</p>
<p># How many hits before a message is considered spam.<br />
required_score           10.0</p>
<p># Change the subject of suspected spam<br />
rewrite_header subject         *****SPAM*****</p>
<p># Encapsulate spam in an attachment (0=no, 1=yes, 2=safe)<br />
report_safe             2</p>
<p># Enable the Bayes system<br />
use_bayes               1</p>
<p># Enable Bayes auto-learning<br />
bayes_auto_learn        1<br />
bayes_file_mode         777</p>
<p># Enable or disable network checks<br />
skip_rbl_checks         0<br />
use_razor2              1<br />
use_dcc                 1<br />
use_pyzor               1</p>
<p># Mail using languages used in these country codes will not be marked<br />
# as being possibly spam in a foreign language.<br />
# - dutch english french german spanish<br />
ok_languages            nl en fr de es</p>
<p># Mail using locales used in these country codes will not be marked<br />
# as being possibly spam in a foreign language.<br />
ok_locales              en<br />
</code></p>
<p>Der Score Treshold kann beliebig gewählt werden, ich habe ihn zunächst hoch gesetzt, um das System zu testen und werde ihn nach weiteren Analysen der eingehenden eMails noch weiter anpassen. Mittlerweile arbeite ich sehr gut mit einem Score von 8.6. Diesen Wert habe ich gewählt, da ich einen Newsletter beziehe, der leider mit 8.5 Spam-Punkten bewertet wird.</p>
<p>Jetzt machen wir uns an die Bearbeitung einer geeigneten .procmailrc Datei, die procmail steuert und so die Mails je nach Spameinstufung in Mailverzeichnisse sortieren kann. Dazu erstellen wir im HOME-Verzeichnis des Users, unter dem getmail läuft (hier: bjoumail), eine Datei namens .procmailrc mit folgendem Inhalt:<br />
<code># SpamAssassin sample procmailrc<br />
# ==============================</p>
<p># The following line is only used if you use a system-wide /etc/procmailrc.<br />
# See procmailrc(5) for infos on what it exactly does, the short version:<br />
#  * It ensures that the correct user is passed to spamd if spamc is used<br />
#  * The folders the mail is filed to later on is owned by the user, not<br />
#    root.<br />
DROPPRIVS=yes</p>
<p># Your procmail logfile will grow rapidly so remember to comment out<br />
# the first line "LOGFILE=$HOME/.procmailrc.log" once you've verified<br />
# that procmail is performing as expected</p>
<p>LOGFILE=$HOME/.procmailrc.log<br />
VERBOSE=yes<br />
COMSAT=no<br />
MAILDIR=/home/bjoumail/mail<br />
DEFAULT=$MAILDIR<br />
PATH=/usr/local/bin:/usr/bin</p>
<p># Pipe the mail through spamassassin (replace 'spamassassin' with 'spamc'<br />
# if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| /usr/local/bin/spamc -s 256000 -u bjoumail</p>
<p># All mail tagged as spam (eg. with a score higher than the set threshold)<br />
# is moved to a Spam Folder<br />
:0:<br />
* ^X-Spam-Status: Yes<br />
/home/bjoumail/mail/.Spam/</p>
<p># Work around procmail bug: any output on stderr will cause the "F" in<br />
"From"<br />
# to be dropped.  This will re-add it.<br />
# NOTE: This is probably NOT needed in recent versions of procmail<br />
:0<br />
* ^^rom[ ]<br />
{<br />
  LOG="*** Dropped F off From_ header! Fixing up. "</p>
<p>  :0 fhw<br />
  | sed -e '1s/^/F/'<br />
}</p>
<p>#---------------------------------------------------------------------#<br />
# (part 2)<br />
# Add your own filtering commands here, if you want.</p>
<p>#---------------------------------------------------------------------#<br />
# (part 3) How to set up vacation message<br />
#   1. No need to set up .forward file if you use this method.<br />
#   2. Prepare .vacation.msg in your home directory.<br />
#   3. Run "vacation -I" command.<br />
#   4. Uncomment the 2 lines below (Did you replace your_loginname with yours?)<br />
#   5. Remember to comment out the 2 lines when you come back from vacation.</p>
<p>#:0 c<br />
#| /usr/bin/vacation your_loginname<br />
</code></p>
<p><strong>Anmerkungen</strong>: Den Workaround habe ich nicht getestet, sondern einfach so aus einer Vorlage übernommen. Den Folder "Spam" müssen wir in unserem eMailprogramm, welches auf den dovecot IMAP-Server zugreift, natürlich noch erstellen. Da dovecot nicht mit mboxes, sondern mit Maildirs arbeitet, ist der trailing slash bei <strong>/home/bjoumail/mail/.Spam/</strong> besonders wichtig.</p>
<p>Nun muss noch die <em>/usr/local/etc/postfix/master.cf</em> angepasst werden. Dazu ändert mann die Zeile</p>
<p><code>smtp      inet  n       -       n       -       -       smtpd</code></p>
<p>in</p>
<p><code>smtp      inet  n       -       n       -       -       smtpd -o content_filter=spamassassin</code></p>
<p>Man fügt also einen content_filter hinzu. Am Ende der Datei muss man postfix nun mittweilen, was dieser Filter tun soll:</p>
<p><code>spamassassin<br />
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;          unix  -       n       n       -       -       pipe<br />
&nbsp;&nbsp;&nbsp;&nbsp;   user=nobody argv=/usr/local/bin/spamc -e /usr/local/sbin/postfix -oi -f ${sender} ${recipient}</code></p>
<p>Dabei ist sichrzustellen, dass die drei obigen zeilen entweder in einer Zeile in der Datei stehen, oder die zwei unteren Zeilen mit Leerzeichen eingeleitet werden. Der Uer 'nobody' ist ggf. anzupassen.<br />
Jetzt kann man das System testen. Eingehende Mails sollten im Header nun stehen haben, dass sie erfolgreich durch spamassassin gepiped und analysiert wurden und je nachdem als Spam oder Ham eingestuft wurden:</p>
<p>Ham:<br />
<code>X-Spam-Checker-Version: SpamAssassin 3.1.1 (2006-03-10) on<br />
	my-server.domain<br />
X-Spam-Level:<br />
X-Spam-Status: No, score=-2.6 required=10.0 tests=BAYES_00 autolearn=ham<br />
	version=3.1.1</code></p>
<p>Spam:<br />
<code><br />
X-Spam-Flag: YES<br />
X-Spam-Checker-Version: SpamAssassin 3.1.1 (2006-03-10) on<br />
	my-server.domain<br />
X-Spam-Level: ********************************<br />
X-Spam-Status: Yes, score=32.9 required=10.0 tests=BAYES_99,<br />
	DATE_IN_FUTURE_03_06,FROM_ENDS_IN_NUMS,FUZZY_PHARMACY,<br />
	HTML_IMAGE_ONLY_24,HTML_MESSAGE,INFO_TLD,RCVD_NUMERIC_HELO,<br />
	UPPERCASE_25_50,URIBL_AB_SURBL,URIBL_JP_SURBL,URIBL_OB_SURBL,<br />
	URIBL_SBL,URIBL_SC_SURBL autolearn=spam version=3.1.1<br />
</code></p>
<p>Hat man noch einen Ordner voll mit Spam, so kann (und sollte) man Spamassassin trainieren, um Mails besser klassifizieren zu können. Das geht mit sa-learn, bspw<br />
<code>sa-learn &#45;&#45;progress &#45;&#45;spam /home/bjoumail/mail/.Spam/cur/</code><br />
Der Unterordner cur ist der Ordner, wo dovecot die (gelesenen) Mails speichert. Mit dem Trigger --ham kann man spamassassin natürlich auch auf gewollte Mails trainieren. Mehr dazu über <em>man sa-learn</em>.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.bjou.de/blog/2006/06/getmail-dovecot-postfix-spamassassin-und-procmail/feed/</wfw:commentRss>
		<slash:comments>1</slash:comments>
		</item>
		<item>
		<title>MRTS &#8211; MRTG RRDtool Total Statistics</title>
		<link>http://www.bjou.de/blog/2006/03/mrts-mrtg-rrdtool-total-statistics/</link>
		<comments>http://www.bjou.de/blog/2006/03/mrts-mrtg-rrdtool-total-statistics/#comments</comments>
		<pubDate>Wed, 15 Mar 2006 23:55:19 +0000</pubDate>
		<dc:creator>bjou</dc:creator>
				<category><![CDATA[Geek Talk]]></category>
		<category><![CDATA[IT-Security]]></category>
		<category><![CDATA[Tutorials]]></category>

		<guid isPermaLink="false">http://www.bjou.de/blog/2006/03/mrts-mrtg-rrdtool-total-statistics/</guid>
		<description><![CDATA[This little guide is not supposed to be an introduction to SNMP, RRDtool or MRTG though I will give a very tiny look into these programs. The true intension of this tutorial is a properly configured MRTS that, of course, goes along with a correct installation of MRTG, RRDtool and SNMP. So most of the [...]]]></description>
			<content:encoded><![CDATA[<p>This little guide is not supposed to be an introduction to SNMP, RRDtool or MRTG though I will give a very tiny look into these programs. The true intension of this tutorial is a properly configured MRTS that, of course, goes along with a correct installation of MRTG, RRDtool and SNMP. So most of the time we will have to configure these three programs. Moreover this tutorial is based on a FreeBSD system, i am running 6-STABLE. I presume that you have Apache installed and configured properly for the use with PHP. If not, do so! I won&#8217;t explain that here. The following chapter will give a short overview over the programs we need for setting up MRTS. The programm summaries are the ones that can be found on the corresponding official websites along with other detailed information on that specific program. So let’s get started.<span id="more-145"></span></p>
<div align="center">
<!--adsense-->
</div>
<li>
<h3><b><u>Overview</u></b></h3>
</li>
<ul>
<li><strong><u>So what is MRTS anyway?</u></strong></li>
<li><strong>Website</strong>: <a href="http://apt-get.dk/mrts">http://apt-get.dk/mrts</a></li>
<li><strong>Description</strong>: MRTS is short for “MRTG RRDtool Total Statistics”. It is a PHP script written by Thor Dreier that uses &#8220;MRTG and RRDtool to sum up total traffic monthly and yearly&#8221;. But why making words, if you can see the full power of MRTS in some examples: <a href="http://mrts.domainnet.dk">http://mrts.domainnet.dk</a> Just click on any device and MRTS will show you the traffic-graphs and some statistics on how much traffic you have made during the actual/last months/year.</li>
</ul>
<p>OK, now we know what MRTS does, but…</p>
<ul>
<li><strong><u>What is MRTG?</u></strong></li>
<li><strong>Website</strong>: <a href="http://oss.oetiker.ch/mrtg/">http://oss.oetiker.ch/mrtg/</a></li>
<li><strong>Description</strong>: &#8220;The Multi Router Traffic Grapher (MRTG) is a tool to monitor the traffic load on network-links. MRTG generates HTML pages containing graphical images which provide a LIVE visual representation of this traffic. [...] MRTG is based on Perl and C and works under UNIX and Windows NT.&#8221; Again, we want to see some results on that, to see its power. So just take a look at <a href="http://www.stat.ee.ethz.ch/mrtg">http://www.stat.ee.ethz.ch/mrtg</a> and pick a device. I think you’ll be satisfied. MRTG goes along alone very well, so if the examples, you have just seen, is everything you want, just install MRTG and you are free to go. (Note: You must NOT use the mrtg.cfg we will configure below. Just skip the Logformat: rrdtool command for MRTG standalone setup). I, for my part, was not fully satisfied with MRTG, because I wanted my router/server to show me detailed traffic summaries over the last months/year. And that’s where MRTS comes into play, because MRTS sums up this traffic using MRTG and RRDtool.</li>
</ul>
<ul>
<li><strong><u>What is RRDtool?</u></strong></li>
<li><strong>Website</strong>: <a href="http://oss.oetiker.ch/rrdtool/">http://oss.oetiker.ch/rrdtool/</a></li>
<li><strong>Description</strong>: &#8220;If you know MRTG, you can think of RRDtool as a reimplementation of MRTGs graphing and logging features. Magnitudes faster and more flexible than you ever thought possible. RRD is the Acronym for Round Robin Database. RRD is a system to store and display time-series data (i.e. network bandwidth, machine-room temperature, server load average). It stores the data in a very compact way that will not expand over time, and it presents useful graphs by processing the data to enforce a certain data density. It can be used either via simple wrapper scripts (from shell or Perl) or via frontends that poll network devices and put a friendly user interface on it.&#8221; RRDtool allows you to almost log everything and convert it into nice impressive graphs. Take a look at <a href="http://oss.oetiker.ch/rrdtool/gallery/index.en.html">http://oss.oetiker.ch/rrdtool/gallery/index.en.html</a> for some examples.</li>
</ul>
<p>As MRTG needs the Simple Network Management Protocol (SNMP) to work properly, we will take a short look on what this does:</p>
<ul>
<li><strong><u>What is SNMP?</u></strong></li>
<li><strong>Website</strong>: <a href="http://net-snmp.sourceforge.net/">http://net-snmp.sourceforge.net/</a></li>
<li><strong>Description</strong>: &#8220;SNMP is an application-layer protocol that facilitates the exchange of management information between network devices. It is part of the Transmission Control Protocol/Internet Protocol (TCP/IP) protocol suite. SNMP enables network administrators to manage network performance, find and solve network problems, and plan for network growth.&#8221; For our plans, we need to install net-SNMP (formerly known as ucd-SNMP) that consists of various tools relating to SNMP.</li>
</ul>
<p>So, enough explanations, let&#8217;s get to work!</p>
<li>
<h3><u><b>Installing and Compiling</b></u></h3>
</li>
<ul>
	First thing to do is to check, which of the above mentioned packages you already have installed. No idea? Well, then you should do a simple<br />
	<code>ls /var/db/pkg</code> or 	<code>pkg_info</code><br />
	which lists the packages, you have installed using the FreeBSD ports. If you don&#8217;t know what the FreeBSD ports are, then you probably won&#8217;t have to read on, as some basics really are required in this tutorial <img src='http://www.bjou.de/blog/wp-includes/images/smilies/icon_wink.gif' alt=';)' class='wp-smiley' />  Anyway, i&#8217;m trying to explain step by step. After you have checked your installed packages, install the packages you still need.</p>
<p>	<code><br />
	<strong>Net-SNMP</strong>: /usr/ports/net-mgmt/net-snmp<br />
	<strong>MRTG:</strong>     /usr/ports/net-mgmt/mrtg<br />
	<strong>RRDtool</strong>:  /usr/ports/net/rrdtool<br />
	</code></p>
<p>	Some of these programs depend on various libraries (e.g. zlib, libpng, GD), but that&#8217;s the good thing: The port-install-mechanism will automatically check, if you have them or not and it will install them, if needed. Do a<br />
	<code><br />
	make &#038;&#038; make install &#038;&#038; make clean</code></p>
<p>	in the corresponding port directories to install that specific port.<br />
	Now it could take a while until all ports are compiled.
</ul>
<li>
<h3><u><b>Configuring</b></u></h3>
</li>
<ul>
	Alright, as far as I see, we are done compiling <img src='http://www.bjou.de/blog/wp-includes/images/smilies/icon_smile.gif' alt=':)' class='wp-smiley' />  So let&#8217;s get it on, this is the harder part:</p>
<p></p>
<li><u>Configuration of net-SNMP</u></li>
<p>	Create a snmpd.conf with</p>
<p>	<code>snmpconf -i -g basic_setup</code></p>
<p>	This will start a walktrough asking you some questions about your system. The book &#8220;Absolute BSD&#8221; (&#8220;FreeBSD de Luxe&#8221; in Germany) explains in detail, how to properly set up net-SNMP. I&#8217;m gonna make it short:</p>
<ul>
<li>Type y for configuring &#8220;the information returned in the system MIB group&#8221;</li>
<li>syslocation is a string of the location of your server, e.g. &#8220;my room&#8221;, syscontact is you.</li>
<li>Type y for setting &#8220;the value of the sysServices.0 OID&#8221;</li>
<li>The following questions should be answered with &#8220;1&#8243; (yes) or &#8220;0&#8243; (no). Commonly it is &#8220;0&#8243; for repeater and bridge and &#8220;1&#8243; for IP, TCP, SMTP</li>
<li>Type y for configuring &#8220;the agent&#8217;s access control&#8221;. Then answer the following three questions (read-write user based access, read-only user based access, read-write community access) with n. Answer the next one (read-only community access) with y. You will be asked for a community name. Invent one or take the default: public (NOT RECOMMENDED!) The network to accept this community from should be the loopback (127.0.0.1) or something like 192.168.100.0/24 for example. 0.0.0.0/0 will give anyone access. Don&#8217;t use this! Hit [return] for &#8220;no restriction&#8221; in the next question. Then you are finished, don&#8217;t do another community line. The rest can all be answered with n.</li>
<li>Finally send the SNMPdeamon a Hangup Signal for re-reading the configuration, if SNMP was running while configuring: <code>ps faux | grep snmpd</code> tells you the PID to send <code>kill -HUP theSnmpdPid</code> to.</li>
</ul>
<p>	The snmpd.conf is now stored in /usr/local/share/snmp/. Move it to /usr/local/etc/snmp/snmp.conf (Note: This Setup was very basic and does by far not show the power of net-SNMP. But for our purpose that is all we need right now. You should probably read some literature about it and do a better configuration afterwards. Read <a href="http://net-snmp.sourceforge.net/tutorial/tutorial-5/mrtg/index.html">http://net-snmp.sourceforge.net/tutorial/tutorial-5/mrtg/index.html</a> for more information on how to set up MRTG to monitor disk-space, CPU-load and so on&#8230;)</p>
<p>	OK, next step is to start the services, i.e. snmp and mrtg. You can do this via the start scripts in </p>
<p>	<code>/usr/local/etc/rc.d/</code></p>
<p>	Speaking of start-scripts, make one for MRTG (mrtg.sh) if there is non in the directory, we&#8217;ll need it later. Of course, you only have to do this step, if the installation of MRTG does not have put an autostart script itsself into /usr/local/etc/rc.d/. When I installed it, this was not the case so I had to make my own.</p>
<p>	<code>#!/bin/sh<br />
	/usr/local/bin/mrtg &#45;&#45;logging=/var/log/mrtg.log &#45;&#45;pid-file=/var/run/mrtg.pid /etc/mrtg.cfg</code></p>
<p>	Now start snmpd (e.g. &#8216;/usr/local/etc/rc.d/snmpd.sh start&#8217;). Check if the processess are running!<br />
PS: If you are using one of the lastest FreeBSD Releases (5.x and up) you are advised to start the deamons via /etc/rc.conf at boot-time. Consult some other Information source on how to use the rc-scripts.
</ul>
<div align="center">
<!--adsense#linkblockbreit-->
</div>
<ul>
<li><u>Configuration of MRTG</u></li>
<p>
	OK, snmpd is running, let&#8217;s configure MRTG. The binaries should be localted in /usr/local/bin.  The &#8220;\&#8221; in the following command means, that this should be one line. Type</p>
<p>	<code>cfgmaker &#45;&#45;global 'WorkDir: /usr/home/www/stats' &#45;&#45;global 'Options[_]: growright' \<br />
	&#45;&#45;output /etc/mrtg.cfg community@router.abc.xyz</code></p>
<p>	where <em>WorkDir </em>is the directory, where the images and html pages should be placed (Note: Later on we will have no images and htmlpages in that directory, but one php-script (MRTS) and some *.rrd files containing the RRDtool data). It has to be visible to a webbrowser, of course. You can play with the arguments, this one should be the best for the first time, for detailed documentation check the website or your man-pages. <em>output </em>is the directory, where your config file will be placed. Replace <em>community@router.abc.xyz</em> with your specification, i.e. <em>public@localhost</em>. This should test your devices and have an output like this:</p>
<p>	<code> --base: Get Device Info on public@localhost:<br />
	--base: Vendor Id:<br />
	--base: Populating confcache<br />
	--snpo: confcache public@localhost: Descr rl0 --> 1<br />
	--snpo: confcache public@localhost: Descr lp0 --> 2<br />
	--snpo: confcache public@localhost: Descr ed1 --> 3<br />
	--snpo: confcache public@localhost: Descr lo0 --> 4<br />
	--snpo: confcache public@localhost: Descr tun0 --> 5<br />
	--snpo: confcache public@localhost: Ip 127.0.0.1 --> 4<br />
	--snpo: confcache public@localhost: Ip 192.168.100.100 --> 1<br />
	--snpo: confcache public@localhost: Ip 213.23.58.244 --> 5 [It's a dynamic IP, just for your notice <img src='http://www.bjou.de/blog/wp-includes/images/smilies/icon_wink.gif' alt=';)' class='wp-smiley' /> ]<br />
	--snpo: confcache public@localhost: Type 6 --> 1<br />
	--snpo: confcache public@localhost: Type 34 --> 2<br />
	--snpo: confcache public@localhost: Type 6 --> 3 (duplicate)<br />
	--snpo: confcache public@localhost: Type 24 --> 4<br />
	--snpo: confcache public@localhost: Type 23 --> 5<br />
	--base: Get Interface Info<br />
	--base: Walking ifIndex<br />
	--base: Walking ifType<br />
	--base: Walking ifSpeed<br />
	--base: Walking ifAdminStatus<br />
	--base: Walking ifOperStatus<br />
	--base: Writing /etc/mrtg.cfg<br />
	</code><br />
	If it does not do something similar like this, you did something wrong in configuring snmpd or snmpd is not running. So take a look at the output. What do you want to log? I wanted to log my internet-traffic, so i had to take device number 5 (device 4, i.e., is software-loopback, device 1 is internal traffic in your LAN). Open your newly configured /etc/mrtg.cfg and check it out. It needs some reconfiguration for our needs. First lines without a comment should be</p>
<p>	<code> WorkDir: /usr/home/www/stats/<br />
	Options[_]: growright<br />
	Logformat: rrdtool<br />
	RunAsDaemon: Yes<br />
	Interval: 5<br />
	PathAdd: /usr/local/bin/<br />
	IconDir: http://my.url.com/stats/icons/</code></p>
<p>	I really suggest to read <a href="http://people.ee.ethz.ch/~oetiker/webtools/mrtg/reference.html">http://people.ee.ethz.ch/~oetiker/webtools/mrtg/reference.html </a> for more and detailed information, though i am going to explain the commands above.<br />
	You already know the first two lines. The third one is very important. Without it, MRTG would create the images itsself without using RRDtool. But i like RRDtool images better <img src='http://www.bjou.de/blog/wp-includes/images/smilies/icon_smile.gif' alt=':)' class='wp-smiley' />  Furthermore you would have to do a cronjob for MRTG AND indexmaker, to create your corresponding html-files as well. With the Logformat command MRTG will save all its data in one file called <em>##device##.rrd</em> where ##device## is i.e. localhost. Thanks to that file, we can use RRDtool to evaluate our graphs. <em>RunAsDaemon </em>is a very nice feature for avoiding a cronjob. After starting MRTG, it is being daemonized and not launched repeatedly (as it would have been with cron). MRTG will be active every 5 minutes. <em>PathAdd </em>is not always needed, but it can&#8217;t harm. It&#8217;s the path to your RRDtool binary and <em>IconDir </em>is the directory the icons are in, that came with the installation of MRTG (look for the icons in /usr/local/share/mrtg/) After these commands your devices are being listet. Comment those out that you do not want to log. I just wanted to log device #5. Should look something like this:</p>
<p>	<code>### Interface 5 >> Descr: 'tun0' | Name: '' | Ip: '213.23.58.171' | Eth: '' ###</p>
<p>	Target[localhost_5]: 5:public@localhost:<br />
	SetEnv[localhost_5]: MRTG_INT_IP="213.23.58.171" MRTG_INT_DESCR="tun0"<br />
	MaxBytes[localhost_5]: 750000<br />
	Title[localhost_5]: Traffic Analysis<br />
	PageTop[localhost_5]: &lt;H1&gt;Internet Traffic Analysis&lt;/H1&gt;<br />
	&lt;TABLE&gt;<br />
	&lt;TR&gt;&lt;TD&gt;System:&lt;/TD&gt;     &lt;TD>FreeBSD 6-STABLE in Bjou's Home in Karlsruhe City&lt;/TD&gt;&lt;/TR&gt;<br />
	&lt;TR&gt;&lt;TD&gt;Max Speed:&lt;/TD&gt;  &lt;TD>Arcor-DSL 6 MBit&lt;/TD>&lt;/TR&gt;<br />
	&lt;/TABLE&gt;<br />
	</code></p>
<p>	You are free to alter some values, I deleted some of them within the &lt;TABLE&gt;&lt;/TABLE&gt; because I didn&#8217;t find them very interesting. It will not affect your MRTS appearance anyway, but your MRTG output, if you want to make a standalone MRTG installation or if you want to use an &#8220;on-the-fly&#8221;-script as <a href="http://www.fi.muni.cz/~kas/mrtg-rrd">mrtg-rrd</a> or <a href="http://my14all.sourceforge.net/">14all.cgi</a>  (which I also advise to take a look at, though you don&#8217;t need it). Maybe don&#8217;t alter anything before really getting started, you will know what you want to alter later, when you see the results (<em>Note again</em>: You won&#8217;t actually see the above strings in your MRTS output!). First line is the most important one: 5:public@localhost: 5 is your interface number mentioned above followed by <em>community@router.abc.xyz</em>. Thor Dreier explains on his MRTS website how to get the value for MaxBytes:</p>
<blockquote><p> &#8220;In &#8220;MaxBytes&#8221; we have set how many bytes the device maximum can transfer:<br />
	(100Mbit * 1000000bit/Mbit / 8byte/bit = 12500000byte).&#8221;<br />
	So as I have a 1.5 MBit bandwidth, we&#8217;ll have to set: (6Mbit * 1000000bit/Mbit / 8byte/bit = 750000 byte)</p></blockquote>
<p>	Be sure to read <a href="http://apt-get.dk/mrts">Thor&#8217;s MRTS site</a> properly for some additional information I did not mention here (especially if your are a Debian User).<br />
	What, you are still here? READ IT NOW! <img src='http://www.bjou.de/blog/wp-includes/images/smilies/icon_wink.gif' alt=';)' class='wp-smiley' /><br />
	OK, looks like we configured the mrtg.cfg well, so let&#8217;s start MRTG using our script above. You are getting error messages? No problem, this is normal the first time(s) you start MRTG. You maybe want to run MRTG under a certain user (running processes under root is not always the wisest decision) <img src='http://www.bjou.de/blog/wp-includes/images/smilies/icon_wink.gif' alt=';)' class='wp-smiley' />  therefore you will need the &#45;&#45;user=mrtg_user &#45;&#45;group=mrtg_group switches in your mrtg.sh startscript. It could be that there are some problems then, if mrtg_user does not have the rights it needs to access/write certain files. Try yourself, I won&#8217;t explain that here.<br />
	Having started MRTG, it should say &#8220;Daemonizing MRTG &#8230;&#8221; as a result. Check your logfile and your processes if it was successful.
</ul>
<ul>
<li><u>Configuration of RRDtool</u></li>
<p>
	RRDtool does not really need any further configuration, so I would say, we are almost done. Anyway, check if RRDtool<br />
	at least works. You can do this by taking a look at <A target="_blank" HREF="http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/tutorial/rrdtutorial.html">http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/tutorial/rrdtutorial.html</A><br />
	and doing this lesson at least until you got the first results and you are sure, RRDtool works.
</ul>
<ul>
<li><u>Configuration of MRTS itsself</u></li>
<p>
	Now comes the best part, the wonderful MRTS. Download from <A target="_blank" HREF="http://apt-get.dk/mrts/download/">http://apt-get.dk/mrts/download/</A></p>
<p>	(same files with different extension) and rename to *.php. Open the file and configure the following:</p>
<p>	<CODE><br />
	/* The directory where the rrd files are located */<br />
	$dir = &#8216;/usr/home/www/stats&#8217;;</p>
<p>	/* List all devices that MRTS should&#8217;n display, */<br />
	$exclude = array(&#8217;secret&#8217;, &#8216;topsecret&#8217;);</p>
<p>	/* RRDtool path &#8211; where is the the executable located */<br />
	$rrdcommand = &#8216;/usr/local/bin/rrdtool&#8217;;<br />
	</CODE><P></p>
<p>	The first path is exactly the MRTG WorkDir, second one explains itsself, last one is the path to your rrdtool executable. (<em>Note:</em><br />
	 <em>rrdtool</em> in the path listed above in not a folder, but the rrdtool binary!)<br />
	Well, that&#8217;s it. If your Apache is properly configured, give it a try. But first check, that you do not run your script in &#8220;safe mode php&#8221;.<br />
	If you server does this as standard, you have to put something like this in your httpd.conf: </p>
<p>	<P></p>
<p>	<CODE><br />
	&lt;Directory &#8220;/path/to/mrts&#8221;&gt;<br />
                 php_admin_value safe_mode 0<br />
	&lt;Directory&gt;<BR><br />
	</CODE></p>
<p>	<P><br />
	Put the MRTS php script in <CODE>/usr/home/www/stats/</CODE> and surf on it. And&#8230;.. whoo-hoo, you are done! <img src='http://www.bjou.de/blog/wp-includes/images/smilies/icon_smile.gif' alt=':)' class='wp-smiley' /><br />
</UL></p>
<div align="center">
<!--adsense#anzeigenblocknurtext-->
</div>
]]></content:encoded>
			<wfw:commentRss>http://www.bjou.de/blog/2006/03/mrts-mrtg-rrdtool-total-statistics/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Automatisiertes Fifa-WM2006-Ticket-Polling</title>
		<link>http://www.bjou.de/blog/2006/03/automatisiertes-fifa-wm-ticket-polling/</link>
		<comments>http://www.bjou.de/blog/2006/03/automatisiertes-fifa-wm-ticket-polling/#comments</comments>
		<pubDate>Mon, 06 Mar 2006 18:48:20 +0000</pubDate>
		<dc:creator>bjou</dc:creator>
				<category><![CDATA[Geek Talk]]></category>
		<category><![CDATA[Tutorials]]></category>

		<guid isPermaLink="false">http://www.bjou.de/blog/?p=56</guid>
		<description><![CDATA[Da Flo&#8217;s und mein Vorhaben, doch noch irgendwie an WM-Tickets zu kommen, durch das dämliche FIFA-System vor kurzem jäh zerstört wurde, musste eine Idee her. Laut FIFA wird &#8220;In der bis zum 15. April andauernden Periode [...] das Angebot immer wieder durch neu eingestellte Ticket-Kontingente aktualisiert&#8220;. Ständig selbst die Seite aufzurufen ist umständlich, außerdem muss [...]]]></description>
			<content:encoded><![CDATA[<p>Da Flo&#8217;s und mein Vorhaben, doch noch <strong>irgendwie</strong> an WM-Tickets zu kommen, durch das <a href="http://fluxon.net/archives/111-I-Hate-The-World-Today....html">dämliche FIFA-System</a> vor kurzem jäh zerstört wurde, musste eine Idee her. Laut FIFA wird &#8220;<em>In der bis zum 15. April andauernden Periode [...] das Angebot immer wieder durch neu eingestellte Ticket-Kontingente aktualisiert</em>&#8220;. Ständig selbst die Seite aufzurufen ist umständlich, außerdem muss man Glück haben, zufällig rechtzeitig vorbeizuschauen. In der freeX 1/06 hatte ich ein Shellscript gefunden, welches automatisch Bookmarks auf Änderungen überwacht und dann ne Mail abschickt. Das war allerdings sehr komplex und lief auf meinem System nicht richtig rund. Eine weitere Möglichkeit ist der WM&#8217;2006 Ticket Alarm von mainhattensoftware.de, allerdings ist der nur für Windows und nicht kostenlos. Also musste was eigenes her -> Weiterlesen&#8230;<br />
<span id="more-56"></span><br />
Das folgende Skript geht nach einem einfachen Prinzip vor: FIFA-Seite besorgen und MD5 Hash darüber machen. Stimmt der Hash nicht mehr überein, hat sich etwas an der Seite getan, dann wird der diff-Output an die entsprechenden Personen gemailt. Es sei noch gesagt, dass die Mail von einer dynamischen Adresse aus generiert und abgeschickt wird, d.h. sie ist sehr wahrscheinlich Opfer von Spam-Filtern. web.de oder gmx.de verwerfen diese Mail sofort, sie erscheint nicht mal im Spam-Verdacht. Man sollte sich also eine völlig ungefilterte eMailadresse besorgen. Ein Cronjob der Art</p>
<pre>[code]
# run the ticketwatcher every 15 minutes
*/15 * * * * cd /path/to/ticketwatch &#038;&#038; bash /path/to/ticketwatch/ticketwatch.sh > /dev/null 2>&#038;1
[/code]</pre>
<p> erledigt dann die Automatisierung.</p>
<pre>[code lang="bash"]
#!/usr/local/bin/bash
# Ticketwatch.sh by Bjoern Weiland

URL="http://tickets.fifaworldcup.com/cgi-bin/ds_wmd?fun=pksbyeventreihe&#038;doc=ds_einzel1&#038;affiliate=wmd&#038;key=0"
MAILTO="mail1@mail.com"
COPYTO="mail2@mail.com, mail3@mail.com"  # comma separated list

function init ()
{
        lynx -nolist -dump $URL > lynxdump.old
        md5 lynxdump.old > md5sum.old
}

if ! [ -f lynxdump.old ]; then
        init
fi

lynx -nolist -dump $URL > lynxdump.now
md5 lynxdump.now > md5sum.now

SUMOLD=`tail -c 33 md5sum.old`
SUMNOW=`tail -c 33 md5sum.now`

if [ "$SUMOLD" != "$SUMNOW" ]
        then
                echo Changes!
                diff lynxdump.now lynxdump.old | mail -s Ticketwatch -c $COPYTO $MAILTO
                rm lynxdump.* md5sum.*
                init
        else
                echo No Changes
fi
[/code]</pre>
]]></content:encoded>
			<wfw:commentRss>http://www.bjou.de/blog/2006/03/automatisiertes-fifa-wm-ticket-polling/feed/</wfw:commentRss>
		<slash:comments>4</slash:comments>
		</item>
		<item>
		<title>Printserver in heterogenem LAN mit Turboprint, FreeBSD und einem Canon Pixma IP4200</title>
		<link>http://www.bjou.de/blog/2005/12/printserver-in-heterogenem-lan-mit-turboprint-freebsd-und-einem-canon-pixma-ip4200/</link>
		<comments>http://www.bjou.de/blog/2005/12/printserver-in-heterogenem-lan-mit-turboprint-freebsd-und-einem-canon-pixma-ip4200/#comments</comments>
		<pubDate>Wed, 07 Dec 2005 18:07:30 +0000</pubDate>
		<dc:creator>bjou</dc:creator>
				<category><![CDATA[Geek Talk]]></category>
		<category><![CDATA[Tutorials]]></category>

		<guid isPermaLink="false">http://www.bjou.de/blog/?p=36</guid>
		<description><![CDATA[Dass Canon nicht wirklich Vorreiter ist, wenn es um Druckunterstützung für Linux und *BSD geht, wusste ich schon vor dem Kauf des Druckers, trotzdem wollte ich es mit diesem Gerät versuchen, da es Preis-/Leistungsmäßig doch recht viel hergibt. Auf das Rumgemurkse mit den japanischen Linuxtreibern hatte ich allerdings keine Lust, außerdem kann man so die [...]]]></description>
			<content:encoded><![CDATA[<p>Dass Canon nicht wirklich Vorreiter ist, wenn es um Druckunterstützung für Linux und *BSD geht, wusste ich schon vor dem Kauf des Druckers, trotzdem wollte ich es mit diesem Gerät versuchen, da es Preis-/Leistungsmäßig doch recht viel hergibt. Auf das Rumgemurkse mit den <a href="http://www.linuxforen.de/forums/showthread.php?t=128409">japanischen Linuxtreibern</a> hatte ich allerdings keine Lust, außerdem kann man so die Duplexeinheit sowieso nicht nutzen (geschweigedenn die CD Bedruck-Funktion, aber ob ich die jemals brauchen werde ist ohnehin eine andere Sache). Turboprint soll (leider gegen Lizenz, die sich aber lohnt) sehr gute Treiber für die Pixma Serie bieten, also wollte ich das mal testen. Wunschkonfiguration: FreeBSD 6.0 (ohne X11) als Druckserver mit den Turboprint Treibern. Meine Erfahrungen habe ich zu einem kleinen Howto zusammengefasst, der interessierte Leser möge weiterlesen <img src='http://www.bjou.de/blog/wp-includes/images/smilies/icon_wink.gif' alt=';)' class='wp-smiley' /> </p>
<p>Vorab-Info:  Wer nicht direkt vom Server aus drucken möchte, kann darauf verzichten, Turboprint auf diesem zu installieren, dann reicht es, eine RAW-Queue über CUPS einzurichten und die Treiber auf den Clients zu installieren.<br />
<span id="more-36"></span></p>
<div align="center"<br />
<!--adsense#linkblockbreit-->
</div>
<p>Allein die Installation von Turboprint (TP) gestaltete sich auf FreeBSD allerdings als nicht intuitiv.</p>
<ul>
<li>Vorbereitung: <a href="http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/linuxemu-lbc-install.html">Linux Emulation</a> und <a href="http://wiki.bsdforen.de/index.php/FreeBSD_-_CUPS">CUPS </a>installieren. Da der Pixma IP4200 ein USB Drucker ist, müssen außerdem folgende Einträge im Kernel vorhanden sein (gegebenenfalls neu bauen):<br />
<code><br />
device scbus<br />
device da<br />
device pass<br />
device uhci     # USB Hub, (kann auch ohci sein) siehe unten!<br />
device usb<br />
device umass<br />
device ulpt      # USB Printer device<br />
</code></p>
<blockquote><p>
 Note: To determine whether you need device uhci or device ohci try checking dmesg:</p>
<p>    hivemind# dmesg | grep uhub<br />
    uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
</p></blockquote>
</li>
<li>Turboprint Installationsskripte anpassen: Setupfehler der Art
<p><code>Begin installation now (y/n)? y<br />
setup: lib/install-static: not found<br />
setup: lib/install-post: not found<br />
setup: lib/install-info: not found</code></p>
<p>habe ich behoben, indem ich in den genannten Dateien folgende Änderung vorgenommen habe:<br />
<code> #!/bin/bash --> #!/usr/local/bin/bash</code> (#!/bin/sh sollte es auch tun). Diese Ersetzungen kann man am besten gleich bei allen Dateien machen, die die Shell brauchen, insbesondere auch bei lib/pstoturboprint.</li>
<li>Sollten Fehler über <code>illegal arguments</code> von <code>install</code> erscheinen, dann folgende Ersetzungen in der lib/install-static vornehmen:<br />
<code>install &#45;&#45;mode=644 --> install -m 644</code></li>
<li>Da bei mir weder ein &#8216;lp&#8217; user noch eine &#8216;lp&#8217; Gruppe existierte, musste ich diese Zeilen <code><br />
chown lp "$TPPATH_LOG/turboprint_lpr.log"<br />
chown lp "$TPPATH_LOG/turboprint_cups.log"<br />
chgrp lp "$TPPATH_LOG/turboprint_lpr.log"<br />
chgrp lp "$TPPATH_LOG/turboprint_cups.log"</code><br />
in der install-post noch entsprechend umändern.</p>
</li>
<p>Danach funzte die Installation mit <code>sh setup</code> einwandfrei. Die Erst-Konfiguration konnte ich nun mit tpsetup vornehmen, da mir für xtpsetup ja das nötige X-Paket (bzw die gtk-libs) fehlen. Bei der Installation sollte CUPS erkannt werden (&#8220;Installation for CUPS printing system (TP_CUPS=1)&#8221;). Das war allerdings nicht der Fall (&#8220;TP_CUPS=0&#8243;), also musste ich folgende Einträge in der system.cfg von Turboprint anpassen. Dies sind die angepassten Pfade:</p>
<p><code>TPPATH_CUPSDRIVER=/usr/local/share/cups/model<br />
TPPATH_CUPSSETTINGS=/usr/local/etc/cups/ppd<br />
TPPATH_CUPSFILTER=/usr/local/libexec/cups/filter<br />
</code></p>
<p>Gegebenenfalls nochmals setup ausführen. Die Installation solte jetzt ohne weitere (Fehler-)meldungen ablaufen (so war&#8217;s zumindest bei mir).</p>
<li>Turboprint ist nun installiert und kann angepasst werden: tpsetup ausführen für initiale Konfiguration, den Canon Drucker hinzufügen. Am Ende sollte das Ganze z.B. so aussehen:<br />
<code><br />
=============================================<br />
Turboprint Setup - Printer Setup
<pixmaip4200>
=============================================<br />
#   Print Jobs   : 0<br />
S - Short Name   : PixmaIP4200<br />
L - Long Name    : Canon_PIXMA_iP4200<br />
C - Connection   : Local Printer<br />
N - Device Name  : usb:/dev/ulpt0<br />
 ...snip...</pixmaip4200></code></li>
<li>Wenn nicht schon geschehen, sollte man nun die Datei <code>/usr/local/etc/cups/cupsd.conf</code> anpassen. Dies ist sehr wichtig und muss fehlerfrei geschehen. Ich habe Ewigkeiten gebraucht, rauszufinden, warum der Drucker korrekt erkannt wird, aber einfach nicht drucken will. Es lag ein kleiner Fehler in der cupsd.conf vor, was schwer in der Logfile einzusehen war und den Druck verhinderte. Ich habe die Datei fast so gelassen wie sie kam, habe nur Logfiles, Servername und -admin gesetzt, jedoch die <code>Security Options</code>-Sektion detailliert bearbeitet, hierfür sollte man sich Zeit lassen.</li>
<li>Über das CUPS Webinterface kann man nun auf die CUPS Drucker zugreifen:<br />
<code>http://server-ip:631</code><br />
Als nächstes sollte der neue Drucker hinzugefügt werden. Da sich Turboprint mittlerweile sehr gut in CUPS integriert, sollten die TP-Treiber hier nun verfügbar sein. Zunächst trägt man einen Namen und eine Beschreibung ein. Unter &#8216;Device for PixmaIP4200&#8242; wählt man nun &#8216;USB Printer #1&#8242;. Dieser Eintrag ist allerdings nur verfügbar, wenn das nötige device /dev/ulpt0 existiert, was wiederum nur mit der oben genannten Kerneloption der Fall ist. Nun wählt man den &#8216;Canon Turboprint&#8217; Treiber und das entsprechende Modell. Der Drucker sollte nun erkannt werden. </li>
<li>Um die Installation zu testen, können wir nun Testseiten drucken. Ganz intuitiv unter <code>http://server-ip:631/printers/PixmaIP4200</code> und/oder direkt mit <code>tpconfig</code>: Drucker auswählen, in die Toolbox wechseln (T) und eine Testpage (T) auswählen und drucken (0 &#8211; 2). <em>Bemerkung:</em> PixmaIP4200 ist der gewählte Druckername.</li>
<div align="center"<br />
<!--adsense#anzeigenblocknurtext-->
<p />
</div>
<li>Sollte alles funktionieren, kann der Drucker freigegeben werden für das lokale Netz. Das geht z.b. per IPP/CUPS oder Samba. Dabei helfen:<br />
<a href="http://wiki.bsdforen.de/index.php/FreeBSD_-_Drucken_in_heterogener_Umgebung_mit_CUPS">Drucken über Samba</a><br />
<a href="http://wiki.unixboard.de/index.php/Drucken_mit_Windows">Drucken über IPP</a><br />
Ich habe das ganze über IPP gelöst, geht einwandfrei schnell und ohne Probleme beim ersten Mal (bei <strong>Windows!!</strong>)</li>
<li>Dazu sei noch folgendes zu sagen: Will man in heterogener Umgebung (d.h. einem LAN mit Win und Linux/*BSD Rechnern) drucken, kann der Drucker als RAW-Printer eingerichtet werden, d.h. ohne Treiber. Nachteil: Vom Server direkt aus ist das Drucken unmöglich. Die Treiber werden dann auf jedem Client einzeln installiert, bei einem &#8220;nur-Windows-LAN&#8221; ist das auch kein Problem, da der Pixma mit Windows-Treibern kommt. Windows ist es allerdings egal, ob der Drucker auf dem Server dazu als RAW-Printer eingerichtet ist oder nicht. Wir haben den Drucker auf dem Server nicht als RAW-Printer installiert, sondern mit den TP-Treibern. Vorteil: Man kann vom Server aus drucken und von allen Win-Clients. Nachteil: Man kann nicht von Linux-Rechnern drucken, die auch noch im Netz hängen. Dazu muss man den Drucker wie erwähnt zu nem RAW machen. Das Ganze geschieht wieder über das CUPS-Webinterface, Drucker modifizieren und bei &#8216;Hersteller/Modell&#8217; den Eintrag &#8216;RAW&#8217; auswählen. Jetzt kann man nicht mehr vom Server aus, dafür von allen Clients drucken, ob Win oder Linux/*BSD. Ich habe noch einen ubuntu-Rechner, auf dem nun also das Drucken über den Server ermöglicht werden soll.</li>
<li>Das hat auch eine Weile gedauert, weil es zuerst nicht so funktionierte wie es sollte. Das <a href="http://wiki.ubuntuusers.de/CUPS">ubuntu Wiki</a> sagt z.B., in der client.conf solle der ServerName Eintrag angepasst werden. Das hat bei mir allerdings nicht geklappt, weil dann der Eintrag unter System &#8211; Systemverwaltung &#8211; Drucker den Drucker zwar erkannte, aber immer die Einstellungen des Servers übernahm, also auch, dass der Drucker am lokalen USB-Port hängt, was für den ubuntu Rechner ja nicht zutrifft. Man konnte es zwar in einen IPP Eintrag ändern, jedoch wurde das beim Beenden nicht gespeichert. Der Eintrag in der client.conf blieb bei mir deshalb auskommentiert, ebenso wie das Polling in der cupsd.conf. Die Vorgehensweise war also folgende: TP auf ubuntu installieren, Drucker unter System &#8211; Systemverwaltung &#8211; Drucker hinzufügen (die TP Treiber sollten in das Treiber-Auswahl-Menü nun integriert sein), Pixma4200 Treiber von Canon (Turboprint) installieren und wie bei den Win Rechnern den gleichen IPP-Pfad. Nun sollte alles funktionieren!</li>
<li>Fehlersuche: Sollte etwas nicht klappen, ist die Fehlersuche hier recht schwierig. Es sei zu sagen, dass evtl ein paar zusätzliche Pakete (z.B. für Epson oder HP Drucker) installiert werden müssen/sollten. Auch das Paket a2ps kann nicht schaden, ob man es braucht entzieht sich meiner Kenntnis, ich bin erstmal froh dass alles läuft.
</li>
</ul>
]]></content:encoded>
			<wfw:commentRss>http://www.bjou.de/blog/2005/12/printserver-in-heterogenem-lan-mit-turboprint-freebsd-und-einem-canon-pixma-ip4200/feed/</wfw:commentRss>
		<slash:comments>4</slash:comments>
		</item>
	</channel>
</rss>
