ulogd2-mysql patch for NFCT

Filed in Linux | Networking Leave a comment

Seems ulogd2-mysql table schema is outdated, even in git repo is not updated (but pgsql does … WTH!). So it not log conntrack event to database because missing ct_event fields.

So this is [patch version], already submitted upstream and wait for update.

In real world use. logging conntrack event to database is not really good solution because it grows very (very (very)) fast.
By default ulogd use hashing so it only insert connection on DESTROYED state unless to want it to log every events.

In case you log all events, it hard to know when the connection starts and end because it log NEW and DESTROY event separately.
Matching require connection ID which not unique after system restart.

And if you need for only logging not analyze, keep default hashing on (log only DESTROY event) and just DROP ALL INDEXES.
It slow down and quite consume the valueable disk space.

So i have some projects myself for implement real-time selective accounting of conntrack flows which also can do hashing
and also do logging, statistics, etc (For … interface with SNMP). The idea was to use libnfconntrack and netlink. But i need to find good bindings in Perl first.

UPDATE:
There is another NET::Netfilter::Netflow perl that directy use conntrack-tools for conntrack events to export Netflow v5 to netflow collector but lacks support of IPv6.

Why outdated Perl languages ? … It fast and works well with parsing something!

, , , ,

Iptables for common DNS amplification attack on recursive DNS inside your network.

Filed in Linux | Networking Leave a comment

There are a lots of DNS amplification attack now. In my network with hosts around a few hundreds of servers, my log was never stop.

What is it ? Basically it is a DDoS technique by use large reply of DNS resolving to DDoS target with hugh amount of bandwidth.
This accomplished by spoof query with source IP of the target victim to ask for (large) DNS record, such as ANY reply of ROOT record or isc.org which is most commonly found. Request is usually 60-70 bytes while reply is as much as 2-3K. (That’s why its call amplification). It not only make your network participate in attacks, but also consume your bandwidth.

More details [here]

But it indeed have some basic iptables rules that block most if it if you have some of open DNS resolver inside your network. Or if you need
to have DNS servers that response to large number of clients (i.e. ISP) you should apply this.

This is tcpdump output of the commonly found attack packets for ROOT record and isc.org record.

11:59:09.598925 IP 174.19.x.x.26252 > 202.162.x.x.domain: 39595+ [1au] ANY? . (28)
        0x0000:  4500 0038 3f1f 0000 f311 bd0b ae13 0490
        0x0010:  caa2 4e44 668c 0035 0024 0000 9aab 0100
        0x0020:  0001 0000 0000 0001 0000 ff00 0100 0029
        0x0030:  2328 0000 0000 0000

12:00:29.038230 IP 5.135.x.x.25345 > 202.162.x.x.domain: 10809+ [1au] ANY? isc.org. (36)
        0x0000:  4500 0040 1e14 0000 e811 cec3 0587 c6a2
        0x0010:  caa2 4f09 6301 0035 002c 0000 2a39 0100
        0x0020:  0001 0000 0000 0001 0369 7363 036f 7267
        0x0030:  0000 ff00 0100 0029 1000 0000 8000 0000

If you have time to look into DNS request structure. at 0x0028. 0000 ff00 is or Root DNS request, another one is isc.org request.
You can match with iptables with following u32 match rules.

iptables -A INPUT -p udp --dport 53 -m u32 --u32 "0x28=0x0000ff00"
iptables -A INPUT -p udp --dport 53 -m u32 --u32 "0x28=0x03697363 && 0x2c=0x036f7267"
iptables -A INPUT -p udp --dport 53 -m u32 --u32 "0x28=0x02646b00"

Update: add last rules for commonly found ANY query on .dk domain.

This can use to block almost 99% of the malicious query.
But … This example not drop any packages, so apply -j DROP or modify it to suit your needs.
But in some cases you need to have more advance checking to allow legitimate request.

There is another example that is really use in my Linux firewall which utilize iptables recent match to give some thresholds
for legitimate request and block if it too frequent.

iptables -t mangle -A FLOOD_DNS_ADD -j NFLOG --nflog-prefix 'FLOOD_DNS:'
iptables -t mangle -A FLOOD_DNS_ADD -m recent --name FLOOD_DNS_BLOCK --rsource --set -j DROP

iptables -t mangle -A FLOOD_DNS_CHECK -m recent --name FLOOD_DNS_BLOCK --rsource --update --hitcount 1 --seconds 60 -j DROP
iptables -t mangle -A FLOOD_DNS_CHECK -m recent --name FLOOD_DNS_BLOCK --rsource --remove -j RETURN
iptables -t mangle -A FLOOD_DNS_CHECK -m recent --name FLOOD_DNS --rsource --update --hitcount 10 --seconds 60 -j FLOOD_DNS_ADD
iptables -t mangle -A FLOOD_DNS_CHECK -m recent --name FLOOD_DNS --rsource --set -j RETURN

iptables -t mangle -A PREROUTING -p udp --dport 53 -m u32 --u32 "0x28=0x0000ff00" -j FLOOD_DNS_CHECK
iptables -t mangle -A PREROUTING -p udp --dport 53 -m u32 --u32 "0x28=0x03697363 && 0x2c=0x036f7267" -j FLOOD_DNS_CHECK
iptables -t mangle -A PREROUTING -p udp --dport 53 -m u32 --u32 "0x28=0x02646b00" -j FLOOD_DNS_CHECK

 

Do not rely on this protection. Make sure you apply ACL to your caching nameservers or turn recursive lookup off completely if you does not need it (run as authoritative name servers for your domains only). Not going in to details here, Google is your friends 🙂

, , , , ,

Flashcache initscripts

Filed in Linux Leave a comment

Just want to share my initscripts for flashcache.

Download [ here ].

Flashcache initscripts from original git repo is full of crap. And intend for use in only one scenario where it use as writeback and need to be mount. So I decided to rewrite the script to fit my requirements. This script can also use writeback/writethrugh mode and optionally can setup LVM and mount it.

In my setup I use flashcache as libvirt datastore backend based on LVM. The script will setup flashcache in writethough mode then run vgscan and vgchange on the configured volume group.

CAUTION!

IF YOU WANT TO USE LVM PV ON TOP OF FLASHCACHE.
MAKE SURE YOU EXCLUDE BACKEND/CACHE DEVICES FROM LVM SCANNING
BY EDIT /etc/lvm/lvm.conf.

For CentOS, add this line in devices {} section. In this example /dev/sda3 is backend device and /dev/sdb is SSD cache.

filter = [ "r|/dev/sda3|","r|/dev/sdb|" ]

and comment

# filter = [ "a/.*/" ]
ANOTHER CAUTION!

qemu/kvm with flashcache backend have problems about data corruption on flashcache. If data get writes to disk with ‘cache=none’ on qemu/kvm with windows guests. Don’t use cache=none on windows guests. This seems to related to DirectIO or FUA which bypass cache mechanism used by flashcache that causes corruption.

, , , , ,

OpenVPN custom ports and bridge with SELinux enabled

Filed in Linux | Networking Leave a comment

By default, SELinux policy on RHEL/CentOS only allow OpenVPN to work only on port 1194, and also without ability to execute brctl for setup bridge (even you configured script-security 2 in config)

You need to create SELinux plugins module to enable OpenVPN to execute brctl from external script and also able to connect any ports, but not listen (read bottom of this article, there is the way to enable allow listen to specific ports that not already defined by other SELinux policy).

here is how …

First, install required package

  • checkpolicy
  • policycoreutils
  • policycoreutils-python

create folder for store plugins such as /etc/selinux/local, cd to that folder

create openvpn-local.te files

module openvpn-local 1.0;

require {
        type openvpn_t;
        type port_t;
        type brctl_exec_t;
        class file { read getattr open execute execute_no_trans };
        class tcp_socket { name_connect };
}

#============= openvpn_t ==============
allow openvpn_t brctl_exec_t:file { read getattr open execute execute_no_trans };
allow openvpn_t port_t:tcp_socket { name_connect };

then compile module with

# checkmodule -M -m openvpn-local.te -o openvpn-local.mod

Then package selinux module with

# semodule_package -o openvpn-local.pp -m openvpn-local.mod

You can also download my prepackaged plugin here.

Now module is compiled, you can install by run

# semodule -i openvpn-local.pp

From now on, you should able to connect to OpenVPN and can execute brctl for interface setup scripts from OpenVPN.
Actually we can also enable OpenVPN script to execute any binary, but it too risky and not recommended (I will not talk into details here, it can complete by adjust above policy a bit).

But if you want OpenVPN to be able to listen to other port other than 1194. Create policy for listen to any ports is too risk, the safe way is to edit SELinux policy openvpn_t port lists. This archived by using this command to list ports allow by current openvpn SELinux policy.

# semanage port -l | grep openvpn_port_t
openvpn_port_t                 tcp      1194
openvpn_port_t                 udp      1194

This show only tcp/udp 1194. Now add the ports you want, but SELinux only allow port not defined by others type (see output of ‘semanage port -l’ to check which ports already defined).

This command add TCP port 11940 to openvpn_port_t to SELinux Policy

# semanage port -a -t openvpn_port_t -p tcp 11940

Your OpenVPN now should able listen on TCP port 11940.

I usually use this to test captive-portal remotely using VM bridged to OpenVPN interface that bridged to hotspot network on remote side.
This is example of my openvpn configuration that bridge to vlan10 bridge. (I did not include any encryption here, just for testing).

hotspot.conf

port 11940
proto tcp-server
dev-type tap
dev vpn.hotspot
persist-tun
script-security 2
up /etc/openvpn/hotspot.up

hotspot.up (remember to chmod +x)

#!/bin/bash

/sbin/ip link set dev vpn.hotspot up
/usr/sbin/brctl addif vlan10 vpn.hotspot

, , , , , ,

Linux: Prefer IPv4 over IPv6 in dual-stack environment (and prevent problems when only IPv4 exists)

Filed in Linux | Networking Leave a comment

On Linux, there is a feature of IPv6 stack on linux that allow IPv4 mapped as IPv6 address using IPv6 mapped IPv4 address (::ffff/96). This allow IPv6 capable application to use only single socket to accept or connect both IPv4 and IPv6.

By default, if you have both IPv4 and native IPv6. Linux system application (glibc) will use IPv6 by default if it exists because getaddrinfo() will return IPv6 first. This is controlled by glibc’s IPv6 address selection preference for getaddrinfo() in /etc/gai.conf. (Note: some application may have mechanism to decide which address to use other than what provide by getaddrinfo()).

/etc/gai.conf control address selection by label and preferences. This is some of it in Gentoo Linux

# Configuration for getaddrinfo(3).
#
# So far only configuration for the destination address sorting is needed.
# RFC 3484 governs the sorting. But the RFC also says that system
# administrators should be able to overwrite the defaults. This can be
# achieved here.
#
# All lines have an initial identifier specifying the option followed by
# up to two values. Information specified in this file replaces the
# default information. Complete absence of data of one kind causes the
# appropriate default information to be used. The supported commands include:
#
# reload
# If set to yes, each getaddrinfo(3) call will check whether this file
# changed and if necessary reload. This option should not really be
# used. There are possible runtime problems. The default is no.
#
# label
# Add another rule to the RFC 3484 label table. See section 2.1 in
# RFC 3484. The default is:
#
#label ::1/128 0
#label ::/0 1
#label 2002::/16 2
#label ::/96 3
#label ::ffff:0:0/96 4
#label fec0::/10 5
#label fc00::/7 6
#label 2001:0::/32 7
#
# This default differs from the tables given in RFC 3484 by handling
# (now obsolete) site-local IPv6 addresses and Unique Local Addresses.
# The reason for this difference is that these addresses are never
# NATed while IPv4 site-local addresses most probably are. Given
# the precedence of IPv6 over IPv4 (see below) on machines having only
# site-local IPv4 and IPv6 addresses a lookup for a global address would
# see the IPv6 be preferred. The result is a long delay because the
# site-local IPv6 addresses cannot be used while the IPv4 address is
# (at least for the foreseeable future) NATed. We also treat Teredo
# tunnels special.
#
# precedence
# Add another rule to the RFC 3484 precedence table. See section 2.1
# and 10.3 in RFC 3484. The default is:
#
#precedence ::1/128 50
#precedence ::/0 40
#precedence 2002::/16 30
#precedence ::/96 20
#precedence ::ffff:0:0/96 10
#
# For sites which prefer IPv4 connections change the last line to
#
#precedence ::ffff:0:0/96 100
#
# scopev4
# Add another rule to the RFC 3484 scope table for IPv4 addresses.
# The definitions in RFC 3484 are equivalent to:
#
#scopev4 ::ffff:169.254.0.0/112 2
#scopev4 ::ffff:127.0.0.0/104 2
#scopev4 ::ffff:10.0.0.0/104 5
#scopev4 ::ffff:172.16.0.0/108 5
#scopev4 ::ffff:192.168.0.0/112 5
#scopev4 ::ffff:0.0.0.0/96 14
#
# For sites which use site-local IPv4 addresses behind NAT there is
# the problem that even if IPv4 addresses are preferred they do not
# have the same scope and are therefore not sorted first. To change
# this use only these rules:
#
scopev4 ::ffff:169.254.0.0/112 2
scopev4 ::ffff:127.0.0.0/104 2
scopev4 ::ffff:0.0.0.0/96 14

In some network this assumption may cause some problems because most IPv6 connection right now is not as good as IPv4 (more lattency, etc.). Sometimes even cause serious problems because IPv6 link-local (address begin with ff80::) are considered valid IPv6 address. Since current Linux distributions shipped with IPv6 enabled (ipv6 modules loaded or compiled in), some application failed to work if you do not have IPv6 connectivity or at least slow to connect because IPv6 was attempt first and it timed out.

As you see in the comment
If you want to prefer IPv4 over IPv6, just change precedence line of ::ffff:0:0/96 from 10 to 100
(If you do not have /etc/gai.conf at all, create it and add only that line, it should work).

But … Some other default configuration of glibc on some distros are still to have problems with /etc/hosts. Make sure you have this line in [b]/etc/host.conf[/b] if you want dual stack to work as it should. 

multi on
 

This line is require to allow glibc to return multiple addresses when using /etc/hosts, otherwise it will return only the first match. IPv4 or IPv6 only applications will have problems. Let’s say if you have ‘::1 localhost’ first and ‘127.0.0.1 localhost’ second. ping localhost will failed if you not have ‘multi on’ in /etc/host.conf. (This will have performance problems if you have large /etc/hosts, but this is only way until glibc address hinting on /etc/hosts get implemented)

Moreover, many application are not aware that when both IPv4 and IPv6 exists, they don’t do multiple attempts (fallback) using different address family (when glibc return multiple address so). Some times they do but you need to explicitly config it to do that.

One application I found with this problem is [b]postfix[/b] it seems failed to fallback to another address family while multiple address family exists (IPv4 and IPv6). You need to add this line to postfix configuration (main.cf) or your mail may stuck in bounce queue because it cannot connect to another server via IPv6.

smtp_address_preference=any
 

, , , , , ,

TOP