Git Product home page Git Product logo

openwrt-irq-balance's Introduction

OpenWrt路由器多核终端均衡脚本

目的

通过均衡分配用于处理中断的CPU核心,提升系统的吞吐性能

原理

OpenWrt默认、Lean OpenWrt默认、使用irqbalance及转发优化版本的核心分配情况如下(IPQ40xx):

中断 OpenWrt Lean irqbalance 转发优化
网络队列rx CPU123 CPU0123 CPU123 CPU12交替
网络队列tx 交替 交替 交替 CPU12交替
网络中断rx CPU0123 4个一组交替 交替 4个一组CPU12反向交替
网络中断tx CPU0123 交替 交替 CPU12反向交替
无线ahb CPU0123 CPU2 CPU2 CPU3
无线pcie CPU0123 CPU3(设置无效) CPU0 CPU3
其他(usb,dma,gpio...) CPU0123 CPU0123 CPU0/CPU2/CPU3 CPU0

CPU123表示使用CPU1、CPU2、CPU3均可。为了提升局部性以提升缓存效率,中断往往被固定在所有指定CPU中最小的那个,在缺少硬件NAT与千兆网的情况下很容易占满1个CPU核心而其他核心空闲,出现性能瓶颈。因此需要调整中断与CPU的对应关系。

结构

脚本参照init.d的格式编写,可直接复制到路由器的/etc/init.d目录下,调整文件名为set_smp_affinity(如果有同名文件则删除),每次重启后即可自动设置irq与核心的绑定关系。 CPU核心与Mask遮罩值的对应关系如下所示:

CPU0 CPU1 CPU2 CPU3
1 2 4 8

如果需要同时使用多个核心,则直接将遮罩值累加取16进制,如使用CPU2与CPU3则值为C。 项目中脚本与平台的对应关系如下表所示:

脚本名称 对应平台 说明
ipq40xx/openwrt.sh 高通IPQ4018/4019/... 针对有线与无线转发优化的配置
ipq40xx/lean-default.sh 高通IPQ4018/4019/... Lean固件中的默认配置
mt7621/openwrt.sh 联发科mt7621 针对有线与无线转发优化的配置
mt7621/lean-default.sh 联发科mt7621 Lean固件中的默认配置
mt7621/pandorabox.sh 联发科mt7621 适用于潘多拉的有线无线优化配置
bcm53xx/openwrt.sh 博通bcm4708/4709 针对有线与无线转发优化的配置

状态

目前只基于Asus RT-ACRH17测试了IPQ4019平台。

测试场景

使用两台间隔7米的ACRH17组建WDS无线桥接网络,主路由中运行iperf3服务器,主从路由通过QCA9984 5G进行连接,测试电脑连接于从路由的LAN1接口。 两台ACRH17均使用OpenWrt官方19.07.2固件与ath10k无线驱动。 测试电脑中运行代码iperf3 -c 主路由IP -t 30 -P 4。 使用从路由测试均衡脚本。 吞吐量的结果如下表所示:

中断绑定配置 速度/Mbps
OpenWrt默认 495
Lean默认 386
irqbalance 476
转发优化 581

openwrt-irq-balance's People

Contributors

biankasyo avatar presisco avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

openwrt-irq-balance's Issues

Could you update it to fit newest openwrt

please have a look,I build newest openwrt master from offical and get a lot of error log from my ea7500 v1 with your script ,thanks

root@Openwrt:/etc/init.d# /etc/init.d/set_smp_affinity restart
using cpu: 0 1 for network queues
binding cpu for eth0 queues
binding cpu for eth0.2 queues
sh: write error: No such file or directory
binding cpu for eth1 queues
binding cpu for eth1.1 queues
sh: write error: No such file or directory
binding cpu for wlan0 queues
sh: write error: No such file or directory
binding cpu for wlan1 queues
sh: write error: No such file or directory
enabling offload on eth0
enabling offload on eth0.2
enabling offload on eth1
enabling offload on eth1.1
net.core.rps_sock_flow_entries = 8192
set mask 1 for irq: 39
set mask 1 for irq: 40
set mask 1 for irq: 41
set mask 1 for irq: 37
set mask 1 for irq: 38
set mask 1 for irq: 50
set mask 1 for irq: 51

潘多拉(盘古)等使用原厂无线驱动的不完美兼容

感谢分享,脚本效果在 IPQ4019 上很好用
有个小问题,例如 MTK7621 系列,在潘多拉中交换机是 esw,无线是 raX 和 raiX。
稍微改了一下脚本适配原厂驱动

#!/bin/sh /etc/rc.common
# Copyright (C) 2006-2011 OpenWrt.org

START=99
STOP=98

if grep -q 'processor.*: 2' /proc/cpuinfo; then
	eth_core_offset=4
	lan_queue_offset=4
	wan_queue_offset=1
	wifi_core=8
	queue_core_count=2
elif grep -q 'processor.*: 1' /proc/cpuinfo; then
	eth_core_offset=1
	lan_queue_offset=1
	wan_queue_offset=1
	wifi_core=2
	queue_core_count=2
else
	return
fi

usb_core=1

assign_interface_round() {
	local interface=$1
	local cpu_offset=$2
	local count=$3
	
	local index=1
	local cpu=$cpu_offset
	for mask in /sys/class/net/$interface/queues/rx-[0-9]*/rps_cpus
	do
		echo $cpu > $mask
		echo 256 > `dirname $mask`/rps_flow_cnt
		if [ $index -lt $count ]
		then
			cpu=`expr $cpu \* 2`
			index=`expr $index + 1`
		else
			cpu=$cpu_offset
			index=1
		fi
	done
	
	index=1
	cpu=$cpu_offset
	for mask in /sys/class/net/$interface/queues/tx-[0-9]*/xps_cpus
	do
		echo $cpu > $mask
		if [ $index -lt $count ]
		then
			cpu=`expr $cpu \* 2`
			index=`expr $index + 1`
		else
			cpu=$cpu_offset
			index=1
		fi
	done
}

assign_interface() {
	local interface=$1
	local cpu_mask=$2

	for mask in /sys/class/net/$interface/queues/rx-[0-9]*/rps_cpus
	do
		echo $cpu_mask > $mask
		echo 256 > `dirname $mask`/rps_flow_cnt
	done

	for mask in /sys/class/net/$interface/queues/tx-[0-9]*/xps_cpus
	do
		echo $cpu_mask > $mask
	done
}

assign_queues() {
	for netpath in /sys/class/net/eth[0-9]*; do
		eth=`basename $netpath`
		
	done
	
	echo "binding cpu for eth0.1"
	assign_interface_round eth0.1 $lan_queue_offset $queue_core_count
	which ethtool > /dev/null 2>&1 && ethtool -K eth0.1 gro on
	
	echo "binding cpu for eth0.2"
	assign_interface_round eth0.2 $wan_queue_offset $queue_core_count
	which ethtool > /dev/null 2>&1 && ethtool -K eth0.2 gro on
	
	for netpath in /sys/class/net/ra[i0-9]*; do
		wlan=`basename $netpath`
		echo "binding cpu for $wlan"

		assign_interface_round $wlan $lan_queue_offset $queue_core_count
	done

	echo 1024 > /proc/sys/net/core/rps_sock_flow_entries
}

# set net interface queue mask -- /sys/class/net/eth*/queues/rx-*/rps_cpus
set_mask() {
	echo "set mask $2 for irq: $1"
	echo "$2" > "/proc/irq/$1/smp_affinity"
}

set_mask_pattern() {
	local name_pattern="$1"
	local mask="$2"
	
	for irq in `grep "$name_pattern" /proc/interrupts | cut -d: -f1 | sed 's, *,,'`
	do
		set_mask $irq $mask
	done
}

set_mask_index() {
	local name_pattern="$1"
	local index="$2"
	local mask="$3"
	
	set_mask `grep -m$index "$name_pattern" /proc/interrupts | cut -d: -f1 | tail -n1 | tr -d ' '` $mask
}

set_mask_range() {
	local name_pattern="$1"
	local start="$2"
	local end="$3"
	local mask="$4"
	
	local count=`expr $end - $start + 1`
	for irq in `grep "$name_pattern" /proc/interrupts | cut -d: -f1 | head -n$end | tail -n$count | sed 's, *,,'`
	do
		set_mask $irq $mask
	done
}

set_mask_interleave() {
	local name_pattern=$1
	local cpu_offset=$2
	local cpu_count=$3
	local step_size=$4
	
	local step_counter=1
	local cpu_counter=1
	local mask=$cpu_offset
	for irq in `grep "$name_pattern" /proc/interrupts | cut -d: -f1 | sed 's, *,,'`
	do
		set_mask $irq $mask
		if [ $step_counter -eq $step_size ]
		then
			step_counter=1
			if [ $cpu_counter -eq $cpu_count ]
			then
				mask=$cpu_offset
				cpu_counter=1
			else
				mask=`expr $mask \* 2`
				cpu_counter=`expr $cpu_counter + 1`
			fi
		else
			step_counter=`expr $step_counter + 1`
		fi
	done
}

set_mask_interleave_reverse() {
	local name_pattern=$1
	local cpu_offset=$2
	local cpu_count=$3
	local step_size=$4
	
	local step_counter=1
	local cpu_counter=1
	local mask=$cpu_offset
	for irq in `grep "$name_pattern" /proc/interrupts | cut -d: -f1 | sed 's, *,,'`
	do
		set_mask $irq $mask
		if [ $step_counter -eq $step_size ]
		then
			step_counter=1
			if [ $cpu_counter -eq $cpu_count ]
			then
				mask=$cpu_offset
				cpu_counter=1
			else
				mask=`expr $mask / 2`
				cpu_counter=`expr $cpu_counter + 1`
			fi
		else
			step_counter=`expr $step_counter + 1`
		fi
	done
}

# set irq mask -- /sys/irq/*/smp_affinity
set_irq_mask() {
	#ethernet
	set_mask_pattern eth $eth_core_offset
	set_mask_pattern esw $eth_core_offset

	#wifi
	set_mask_pattern ra[i0-9] $wifi_core

	#usb
	set_mask_pattern usb $usb_core
}

start() {
	assign_queues
	set_irq_mask
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.