首页 > *nix技术, 内核技术, 跟踪调试 > 利用pktgen发包工具做性能测试

利用pktgen发包工具做性能测试

2012年2月22日 发表评论 阅读评论 15,254 次浏览

专业的发包仪器非常不错,比如思博伦通信Smartbits测试设备,但唯一的缺点就是太贵,而这个唯一的缺点又不是那么容易克服。还好,伟大的Linux为我们提供了一个先进的发包工具pktgen(http://www.linuxfoundation.org/collaborate/workgroups/networking/pktgenhttp://lxr.linux.no/#linux+v2.6.38.8/Documentation/networking/pktgen.txtftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/),这个工具以内核模块的形式存在,理论上性能应该比同等运行在应用层的工具性能要好,而且还是所谓的多核支持(的确是实打实),下面测试一下(临时用自己家里电脑上的虚拟机搭的环境,以便能够虚拟出需要的多个网卡来,不在意测试结果,而是这个测试过程)。

1,首先加载这个模块,当然需要先确定已经编译了它,加载成功后就会有对应的/proc接口,pktgen是每一个cpu绑定一个内核线程,我的虚拟机有4个cpu,所以这里可以看到4个kpktgend_*文件:

[root@localhost ~]# uname -a
Linux localhost.localdomain 2.6.38.8 #4 SMP Mon Oct 31 20:49:48 CST 2011 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost ~]# cat /usr/src/linux-2.6.38.8/.config | grep CONFIG_NET_PKTGEN
CONFIG_NET_PKTGEN=m
[root@localhost ~]# insmod /usr/src/linux-2.6.38.8/net/core/pktgen.ko 
[root@localhost ~]# cat /proc/cpuinfo | grep processor
processor	: 0
processor	: 1
processor	: 2
processor	: 3
[root@localhost ~]# ls /proc/net/pktgen/
kpktgend_0  kpktgend_1  kpktgend_2  kpktgend_3  pgctrl
[root@localhost ~]# 

2,虚拟机一共有3个网口,拿其中的2个网口(eth5/eth6)用于pktgen测试,设置ip地址,修改速率,:

[root@localhost pktgen]# ifconfig eth5
eth5      Link encap:Ethernet  HWaddr 00:0C:29:97:9B:B4  
          inet addr:192.168.1.95  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe97:9bb4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:15044140 errors:0 dropped:0 overruns:0 frame:0
          TX packets:14210498 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:903846363 (861.9 MiB)  TX bytes:852636332 (813.1 MiB)

[root@localhost pktgen]# ifconfig eth6
eth6      Link encap:Ethernet  HWaddr 00:0C:29:97:9B:BE  
          inet addr:192.168.1.96  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe97:9bbe/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:14256340 errors:0 dropped:0 overruns:0 frame:0
          TX packets:14998657 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:856589260 (816.9 MiB)  TX bytes:899925597 (858.2 MiB)

[root@localhost pktgen]# ethtool -s eth5 autoneg off speed 1000 duplex full
[root@localhost pktgen]# ethtool -s eth6 autoneg off speed 1000 duplex full
[root@localhost pktgen]# ethtool eth5
Settings for eth5:
	Supported ports: [ TP ]
	Supported link modes:   10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Full 
	Supports auto-negotiation: Yes
	Advertised link modes:  1000baseT/Full 
	Advertised pause frame use: No
	Advertised auto-negotiation: Yes
	Speed: 1000Mb/s
	Duplex: Full
	Port: Twisted Pair
	PHYAD: 0
	Transceiver: internal
	Auto-negotiation: on
	MDI-X: Unknown
	Supports Wake-on: d
	Wake-on: d
	Current message level: 0x00000007 (7)
	Link detected: yes
[root@localhost pktgen]# ethtool eth6
Settings for eth6:
	Supported ports: [ TP ]
	Supported link modes:   10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Full 
	Supports auto-negotiation: Yes
	Advertised link modes:  1000baseT/Full 
	Advertised pause frame use: No
	Advertised auto-negotiation: Yes
	Speed: 1000Mb/s
	Duplex: Full
	Port: Twisted Pair
	PHYAD: 0
	Transceiver: internal
	Auto-negotiation: on
	MDI-X: Unknown
	Supports Wake-on: d
	Wake-on: d
	Current message level: 0x00000007 (7)
	Link detected: yes
[root@localhost pktgen]# 

查看两个网口的中断号,并把eth5/eth6做亲和性绑定到特定的cpu(先关闭系统的irqbalance服务):

[root@localhost ~]# cat /proc/interrupts | grep eth
 16:       2586        540    1351588    4172554   IO-APIC-fasteoi   Ensoniq AudioPCI, eth6
 19:       5117    1949714    6098060         40   IO-APIC-fasteoi   eth4, eth5
[root@localhost ~]# /etc/init.d/irqbalance stop
Stopping irqbalance:                                       [  OK  ]
[root@localhost ~]# echo 4 > /proc/irq/19/smp_affinity 
[root@localhost ~]# echo 8 > /proc/irq/16/smp_affinity 
[root@localhost ~]# cat /proc/irq/19/smp_affinity 
04
[root@localhost ~]# cat /proc/irq/16/smp_affinity 
08
[root@localhost ~]# 

3,pktgen测试脚本如下(从文件:ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/examples/pktgen.conf-2-2修改而来),可以看到相当于是把eth5和eth6拿根网线直连起来相互对发数据包:

#! /bin/sh

# FileName: pktgen-eth5-eth6.conf
# modprobe pktgen

function pgset() {
    local result

    echo $1 > $PGDEV

    result=`cat $PGDEV | fgrep "Result: OK:"`
    if [ "$result" = "" ]; then
         cat $PGDEV | fgrep Result:
    fi
}

function pg() {
    echo inject > $PGDEV
    cat $PGDEV
}

# Config Start Here -----------------------------------------------------------


# thread config
# Each CPU has own thread. Two CPU exammple. We add eth1, eth2 respectivly.

PGDEV=/proc/net/pktgen/kpktgend_2
  echo "Removing all devices"
 pgset "rem_device_all" 
  echo "Adding eth5"
 pgset "add_device eth5" 


PGDEV=/proc/net/pktgen/kpktgend_3
  echo "Removing all devices"
 pgset "rem_device_all" 
  echo "Adding eth6"
 pgset "add_device eth6" 


# device config
# delay 0 means maximum speed.

CLONE_SKB="clone_skb 1000000"
# NIC adds 4 bytes CRC
PKT_SIZE="pkt_size 60"

# COUNT 0 means forever
#COUNT="count 0"
COUNT="count 0"
DELAY="delay 0"

PGDEV=/proc/net/pktgen/eth5
  echo "Configuring $PGDEV"
 pgset "$COUNT"
 pgset "$CLONE_SKB"
 pgset "$PKT_SIZE"
 pgset "$DELAY"
 pgset "dst 192.168.1.96" 
 pgset "dst_mac 00:0C:29:97:9B:BE"

PGDEV=/proc/net/pktgen/eth6
  echo "Configuring $PGDEV"
 pgset "$COUNT"
 pgset "$CLONE_SKB"
 pgset "$PKT_SIZE"
 pgset "$DELAY"
 pgset "dst 192.168.1.95" 
 pgset "dst_mac 00:0C:29:97:9B:B4"

# Time to run
PGDEV=/proc/net/pktgen/pgctrl

 echo "Running... ctrl^C to stop"
 pgset "start" 
 echo "Done"

# Result can be vieved in /proc/net/pktgen/eth[5,6]

4,执行上面那个pktgen测试脚本,看数据:

[root@localhost pktgen]# sh pktgen-eth5-eth6.conf 
Removing all devices
Adding eth5
Removing all devices
Adding eth6
Configuring /proc/net/pktgen/eth5
Configuring /proc/net/pktgen/eth6
Running... ctrl^C to stop

另开一个终端,执行mpstat命令查看cpu占用率,看网卡中断的处理是否正常:

[root@localhost ~]# mpstat -P 2,3 1
Linux 2.6.38.8 (localhost.localdomain) 	01/15/2012 	_x86_64_	(4 CPU)

09:13:16 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
09:13:17 AM    2    0.00    0.00   86.87    0.00    0.00   13.13    0.00    0.00    0.00
09:13:17 AM    3    0.00    0.00   74.23    0.00    0.00   25.77    0.00    0.00    0.00
09:13:18 AM    2    0.00    0.00   87.63    0.00    0.00   12.37    0.00    0.00    0.00
09:13:18 AM    3    0.00    0.00   76.53    0.00    0.00   23.47    0.00    0.00    0.00
09:13:19 AM    2    0.00    0.00   85.71    0.00    0.00   14.29    0.00    0.00    0.00
09:13:19 AM    3    0.00    0.00   75.51    0.00    0.00   24.49    0.00    0.00    0.00
09:13:20 AM    2    0.00    0.00   86.60    0.00    0.00   13.40    0.00    0.00    0.00
09:13:20 AM    3    0.00    0.00   70.10    0.00    0.00   29.90    0.00    0.00    0.00
^C
[root@localhost ~]# cat /proc/interrupts | grep eth
 16:       2586        540        588    6664664   IO-APIC-fasteoi   Ensoniq AudioPCI, eth6
 19:       5117        714    9729867         40   IO-APIC-fasteoi   eth4, eth5
[root@localhost ~]# 

执行一段时间后看统计数据:

[root@localhost pktgen]# sh pktgen-eth5-eth6.conf 
Removing all devices
Adding eth5
Removing all devices
Adding eth6
Configuring /proc/net/pktgen/eth5
Configuring /proc/net/pktgen/eth6
Running... ctrl^C to stop
^C
[root@localhost pktgen]# cat /proc/net/pktgen/eth5
Params: count 0  min_pkt_size: 60  max_pkt_size: 60
     frags: 0  delay: 0  clone_skb: 1000000  ifname: eth5
     flows: 0 flowlen: 0
     queue_map_min: 0  queue_map_max: 0
     dst_min: 192.168.1.96  dst_max: 
        src_min:   src_max: 
     src_mac: 00:0c:29:97:9b:b4 dst_mac: 00:0c:29:97:9b:be
     udp_src_min: 9  udp_src_max: 9  udp_dst_min: 9  udp_dst_max: 9
     src_mac_count: 0  dst_mac_count: 0
     Flags: 
Current:
     pkts-sofar: 4405203  errors: 0
     started: 2423250036us  stopped: 2570673239us idle: 61432us
     seq_num: 4405204  cur_dst_mac_offset: 0  cur_src_mac_offset: 0
     cur_saddr: 0x5f01a8c0  cur_daddr: 0x6001a8c0
     cur_udp_dst: 9  cur_udp_src: 9
     cur_queue_map: 0
     flows: 0
Result: OK: 147423202(c147361769+d61432) usec, 4405203 (60byte,0frags)
  29881pps 14Mb/sec (14342880bps) errors: 0
[root@localhost pktgen]# cat /proc/net/pktgen/eth6
Params: count 0  min_pkt_size: 60  max_pkt_size: 60
     frags: 0  delay: 0  clone_skb: 1000000  ifname: eth6
     flows: 0 flowlen: 0
     queue_map_min: 0  queue_map_max: 0
     dst_min: 192.168.1.95  dst_max: 
        src_min:   src_max: 
     src_mac: 00:0c:29:97:9b:be dst_mac: 00:0c:29:97:9b:b4
     udp_src_min: 9  udp_src_max: 9  udp_dst_min: 9  udp_dst_max: 9
     src_mac_count: 0  dst_mac_count: 0
     Flags: 
Current:
     pkts-sofar: 2846965  errors: 0
     started: 2423223982us  stopped: 2570673243us idle: 45963us
     seq_num: 2846966  cur_dst_mac_offset: 0  cur_src_mac_offset: 0
     cur_saddr: 0x6001a8c0  cur_daddr: 0x5f01a8c0
     cur_udp_dst: 9  cur_udp_src: 9
     cur_queue_map: 0
     flows: 0
Result: OK: 147449261(c147403297+d45963) usec, 2846965 (60byte,0frags)
  19308pps 9Mb/sec (9267840bps) errors: 0
[root@localhost pktgen]# 

看起来,这个测试结果也太差了,双向平均才2.5Wpps,后续再拿真实机器测试。

转载请保留地址:http://www.lenky.info/archives/2012/02/1165http://lenky.info/?p=1165


备注:如无特殊说明,文章内容均出自Lenky个人的真实理解而并非存心妄自揣测来故意愚人耳目。由于个人水平有限,虽力求内容正确无误,但仍然难免出错,请勿见怪,如果可以则请留言告之,并欢迎来讨论。另外值得说明的是,Lenky的部分文章以及部分内容参考借鉴了网络上各位网友的热心分享,特别是一些带有完全参考的文章,其后附带的链接内容也许更直接、更丰富,而我只是做了一下归纳&转述,在此也一并表示感谢。关于本站的所有技术文章,欢迎转载,但请遵从CC创作共享协议,而一些私人性质较强的心情随笔,建议不要转载。

法律:根据最新颁布的《信息网络传播权保护条例》,如果您认为本文章的任何内容侵犯了您的权利,请以Email或书面等方式告知,本站将及时删除相关内容或链接。

  1. DamnPoor
    2015年6月9日11:30 | #1

    我抓了下包看它发的是UDP的,在对端收到的时候就丢掉了因为没有监听对应端口的进程.
    这个模块应该只能发UDP的包吧?

  2. 2014年3月14日10:33 | #2

    @Gil
    http://people.kth.se/~danieltt/pktgen/examples/
    作者放到官网上了http://people.kth.se/~danieltt/pktgen/

  3. goter
    2013年7月24日10:39 | #3

    从深信服离职了?不等等上市?

    • lenky
      2013年7月24日14:23 | #4

      深信服是一家非常不错的公司,但我想回长沙了,所以就辞职啦了,:)

  4. lenky
    2013年7月11日12:36 | #5

    @Gil
    我也没有了,之前的在公司办公电脑上,但我已经离职了。
    这个东东已经加入到内核主线了,你可以直接看内核里面的代码吧,比如:
    https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/networking/pktgen.txt?id=refs/tags/v3.10

  5. Gil
    2013年7月11日09:28 | #6

    你好,可否发一份pktgen-testing的examples给我。作者的ftp都是死链,我在网上其他地方都没有找到,我的邮箱是vstwins@21cn.com,十分感谢!

  1. 本文目前尚无任何 trackbacks 和 pingbacks.