sonic-net / sonic Goto Github PK

Sonic-buildimage revision - c199614b69be65edfecc504f6fd3042dbc3d195c

Orchagent aborted with no error messages

Backtrace:

Thread 2 (Thread 0x7efd6c232700 (LWP 50)):
#0  0x00007efd6c314893 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007efd6d23f07e in swss::Select::select (this=this@entry=0x7efd6c231e80, c=c@entry=0x7efd6c231e18, fd=fd@entry=0x7efd6c231dfc, timeout=timeout@entry=4294967295) at select.cpp:77
#2  0x00007efd6d725d24 in ntf_thread () at sai_redis_switch.cpp:37
#3  0x00007efd6cbab970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007efd6d9430a4 in start_thread (arg=0x7efd6c232700) at pthread_create.c:309
#5  0x00007efd6c31b62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

    Thread 1 (Thread 0x7efd6e1f3740 (LWP 45)):
#0  0x00007efd6c268067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007efd6c269448 in __GI_abort () at abort.c:89
#2  0x00007efd6cb55b3d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007efd6cb53bb6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007efd6cb53c01 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007efd6d72762f in ~thread (this=<optimized out>, __in_chrg=<optimized out>) at /usr/include/c++/4.9/thread:146
#6  destroy<std::thread> (this=<optimized out>, __p=<optimized out>) at /usr/include/c++/4.9/ext/new_allocator.h:124
#7  _S_destroy<std::thread> (__p=<optimized out>, __a=...) at /usr/include/c++/4.9/bits/alloc_traits.h:282
#8  destroy<std::thread> (__a=..., __p=<optimized out>) at /usr/include/c++/4.9/bits/alloc_traits.h:411
#9  std::_Sp_counted_ptr_inplace<std::thread, std::allocator<std::thread>, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=<optimized out>) at /usr/include/c++/4.9/bits/shared_ptr_base.h:524
#10 0x00007efd6d7276db in _M_release (this=0xfc23c0) at /usr/include/c++/4.9/bits/shared_ptr_base.h:149
#11 ~__shared_count (this=<optimized out>, __in_chrg=<optimized out>) at /usr/include/c++/4.9/bits/shared_ptr_base.h:666
#12 ~__shared_ptr (this=<optimized out>, __in_chrg=<optimized out>) at /usr/include/c++/4.9/bits/shared_ptr_base.h:914
#13 std::shared_ptr<std::thread>::~shared_ptr (this=<optimized out>, __in_chrg=<optimized out>) at /usr/include/c++/4.9/bits/shared_ptr.h:93
#14 0x00007efd6c26aeaf in __cxa_finalize (d=0x7efd6d93a508) at cxa_finalize.c:56
#15 0x00007efd6d71d983 in __do_global_dtors_aux () from /usr/lib/x86_64-linux-gnu/libsairedis.so.0
#16 0x00007ffd7933b5d0 in ?? ()
#17 0x00007efd6dfeefca in _dl_fini () at dl-fini.c:252
        Backtrace stopped: frame did not save the PC

• jenkins building and uploading docker containers to dockerhub
• jenkins building and uploading to public apt repo setup
• linux kernel build and upload to public apt repo
• network switch hardware accessible to https://sonic-jenkins.westus.cloudapp.azure.com/
• jenkins projects to deploy to public switch hward
• jenkins projects created to execute tests

Will paste the core dump here.

SONiC switches when connected one to another cannot resolve each other because their ports have the same MAC. We need to provide MAC address distribution based on base MAC from device EEPROM.

Here's what is observed when I try to ping one SONiC switch from another:

root@arc-switch1025:/home/admin# ping 20.0.0.5
PING 20.0.0.5 (20.0.0.5) 56(84) bytes of data.
09:04:57.386041 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28
From 20.0.0.4 icmp_seq=1 Destination Host Unreachable
09:04:58.555442 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28
09:04:59.554031 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28
09:05:00.554003 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28
From 20.0.0.4 icmp_seq=2 Destination Host Unreachable
From 20.0.0.4 icmp_seq=3 Destination Host Unreachable
From 20.0.0.4 icmp_seq=4 Destination Host Unreachable
09:05:01.571378 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28
09:05:02.570035 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28
09:05:03.570071 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28
From 20.0.0.4 icmp_seq=5 Destination Host Unreachable
From 20.0.0.4 icmp_seq=6 Destination Host Unreachable
From 20.0.0.4 icmp_seq=7 Destination Host Unreachable
^C
--- 20.0.0.5 ping statistics ---
8 packets transmitted, 0 received, +7 errors, 100% packet loss, time 7031ms
pipe 3
root@arc-switch1025:/home/admin# 09:05:04.587432 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28
09:05:05.586050 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28
09:05:06.586058 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28

Logs from second switch:

09:04:53.329642 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:04:54.329629 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:04:55.387772 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:04:55.387809 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:04:56.385577 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:04:56.385602 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:04:57.385587 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:04:57.385630 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:04:58.554960 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:04:58.554989 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:04:59.553578 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:04:59.553621 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:05:00.553522 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:05:00.553554 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:05:01.570977 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:05:01.571022 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:05:02.569651 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:05:02.569698 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:05:03.569617 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:05:03.569665 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:05:04.587050 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:05:04.587094 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:05:05.585667 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:05:05.585712 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:05:06.585616 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:05:06.585666 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28

Second switch sends ARP reply with same MAC as on first, which cannot be handled correctly.

After mft installation modules are not available
Need to run depmod -a or else mst start fails

This is to ask for 1) a counter for disk/CPU/memory usage ; 2) to limit the var/log folder size on the disk.

Oct 5 09:37:32 CCPSCH01030CCLF NOTICE orchagent: :- setHostIntfsOperStatus: Set operation status DOWN to host interface Ethernet68
Oct 5 09:37:32 CCPSCH01030CCLF NOTICE orchagent: :- doPortTask: Port is set to admin up alias:Ethernet68
Oct 5 09:37:32 CCPSCH01030CCLF ERR syncd: brcm_sai_remove_route:331 L3 route delete failed with error Entry not found (0xfffffff9).
Oct 5 09:37:32 CCPSCH01030CCLF ERR syncd: :- processEvent: failed to execute api: remove: -7
Oct 5 09:37:32 CCPSCH01030CCLF NOTICE syncd: :- exit_and_notify: sending switch_shutdown_request notification to OA
Oct 5 09:37:32 CCPSCH01030CCLF NOTICE syncd: :- exit_and_notify: notification send successfull
Oct 5 09:37:32 CCPSCH01030CCLF ERR orchagent: :- on_switch_shutdown_request: Syncd stopped
Oct 5 09:37:32 sonic INFO docker[8721]: terminate called without an active exception

How source interface is attached to dhcp request in option 82

missing lo address in the redis appdb.

acsadmin@abc:~$ ip addr show dev lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 10.208.95.151/32 brd 10.208.95.151 scope global lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever

127.0.0.1:6379> keys ROUTE_TABLE:10.208.95.151*
(empty list or set)
127.0.0.1:6379>

Neighbor mac could change due to device replacement. Current orchagent does not support neighbor mac changes. Need add support for this.

setup jenkins to publish apt packages to public apt repo
make updates to sonic-mgmt to use the public apt repo
Also need to verify what will be the apt get, will it provide the latest?
kernel deb to be part of MLNX hosted Jenkins and apt repo

document interface application table schema

TASK [test : Check for missing interfaces in SNMP] *****************************
Wednesday 15 March 2017 05:54:42 +0000 (0:00:00.137) 0:01:55.295 *******
failed: [str-s6000-acs-7] => (item=PortChannel01) => {"failed": true, "item": "PortChannel01", "msg": "Minigraph interface PortChannel01 not in SNMP interfaces"}
failed: [str-s6000-acs-7] => (item=PortChannel01) => {"failed": true, "item": "PortChannel01", "msg": "Minigraph interface PortChannel01 not in SNMP interfaces"}
failed: [str-s6000-acs-7] => (item=PortChannel02) => {"failed": true, "item": "PortChannel02", "msg": "Minigraph interface PortChannel02 not in SNMP interfaces"}
failed: [str-s6000-acs-7] => (item=PortChannel02) => {"failed": true, "item": "PortChannel02", "msg": "Minigraph interface PortChannel02 not in SNMP interfaces"}
failed: [str-s6000-acs-7] => (item=PortChannel03) => {"failed": true, "item": "PortChannel03", "msg": "Minigraph interface PortChannel03 not in SNMP interfaces"}
failed: [str-s6000-acs-7] => (item=PortChannel03) => {"failed": true, "item": "PortChannel03", "msg": "Minigraph interface PortChannel03 not in SNMP interfaces"}
failed: [str-s6000-acs-7] => (item=PortChannel04) => {"failed": true, "item": "PortChannel04", "msg": "Minigraph interface PortChannel04 not in SNMP interfaces"}
failed: [str-s6000-acs-7] => (item=PortChannel04) => {"failed": true, "item": "PortChannel04", "msg": "Minigraph interface PortChannel04 not in SNMP interfaces"}

extend the port_config.ini format to have a port alias for each port. add a port alias field in the PORT_TABLE schema. populate port alias field in portsyncd.

This port alias will be used for by snmp subagent as the interface alias.

Hi,

An issue, just to notify you a bunch of repositories among the list in https://github.com/Azure/SONiC/blob/gh-pages/sourcecode.md are not yet opened to the public.

Thanks for opening sonic!

Jan 24 22:50:09 str-msn2700-04 DEBUG teamd_PortChannel32[402]: Ethernet36: lacp info state: 0x3D.
Jan 24 22:50:09 str-msn2700-04 DEBUG teamd_PortChannel40[405]: Ethernet40: lacp info state: 0x3D.
Jan 24 22:50:09 str-msn2700-04 DEBUG teamd_PortChannel48[408]: Ethernet52: lacp info state: 0x3D.
Jan 24 22:50:09 str-msn2700-04 DEBUG teamd_PortChannel40[405]: Ethernet44: lacp info state: 0x3D.
Jan 24 22:50:09 str-msn2700-04 DEBUG teamd_PortChannel16[396]: Ethernet16: lacp info state: 0x3D.

LACP info state is printed out every 1s even when the teamd debug level set to zero.

Inside the bgp container, run the command

# /usr/lib/quagga/zebra --daemon -A 127.0.0.1
% not a local address       

Not sure about the severity, seems a confusing error message to me

Need PTF test for IPv4 decap test

Hi,

While working on a SONiC snap, I hit the hardcoded path to the Redis unix socket. It's defined in header files at build time and ends up in a number of daemons (syncd etc.) and libs (libsairedis). This makes it impractical to relocate SONiC, to have multiple versions of it on the same box or to integrate into a different FS layout.

It would be nice to have a runtime override for the path, either an environment variable (SONIC_REDIS_URL?) or a command-line flag that would have to be passed to each program using Redis (--redis-url).

Thanks,
-- Loïc Minier

syncd docker containers mlnx sdk and it has to match the firmware in order to run. When deploying a new syncd docker with a new version of mlnx sdk, it requires to upgrade the firmware. Currently this is not embed in the syncd docker deployment process in the ansible.

We need to upgrade the firmware when rolling out a new mlnx sdk. We prefer to embed the mlnx firmware inside the syncd docker, and then calling mlnxfwmanager to upgrade the firmware when there is a new firmware.

setup v4/v6 bgp session with dut
advertise 10k v4 + 10k v6 routes to dut
generate packet/verify
testbed setup using ptf docker + vms

C'mon guys, there is mention of a CLA, but not the license this is released under. Can you clear this up please?

Ensure VLAN UP even when no members are UP.

Platform: Mellanox SN2700
Description:
After assigning IP address to router port via ifconfig ping to host from the same subnet is not working. ARP packets are handled, neighbor is added on both sides. ICMP packets on device are sent but not trapped. Possible root cause - trap route for router port is not created (sx_api_router_uc_routes_dump_all.py doesn't show this route). When IP address is set in config and deployed by Ansible ping works as expected.

Hi,

When unpacking SONiC libsaibcm debs, I'm missing libsai.so.1. Indeed, the actual lib – libsai.so.1.0 – is shipped in libsaibcm and libsaibcm-dev includes libsai.so, but I couldn't find libsai.so.1 (might be a mistake on my side?).

Cheers,
-- Loïc Minier

For Tor scanrio, where you have a vlan interface with several ports. When orchagent docker is restarted, all front panel ports are re-created, however those ports are not added back to the vlan interface.

PTF Test
Deploy relay agent - sonic-mgmt
Test execution for nightly build - ansible playbook

The teamd builds a LAG.
After the netdev is removed, and then re-created, the previous teamd process is no longer valid and the previous LAG cannot be re-built.

log:

Ethernet4: Changed port state: "current" -> "expired"
Ethernet0: Changed port state: "current" -> "expired"
carrier changed to DOWN
Ethernet4: Changed port state: "expired" -> "defaulted"
Ethernet0: Changed port state: "expired" -> "defaulted"
Ethernet0: Changed port state: "defaulted" -> "disabled"
Ethernet0: ethtool-link went down.
Loop callback failed with: No such file or directory
ioctl SIOCDELMULTI failed.
Ethernet4: Changed port state: "defaulted" -> "disabled"
Ethernet4: ethtool-link went down.
Loop callback failed with: No such file or directory
ioctl SIOCDELMULTI failed.

acsadmin@xx:~$ ip route show
default via 10.208.80.1 dev eth0 proto zebra

127.0.0.1:6379> hgetall ROUTE_TABLE:0.0.0.0/0

1. "nexthop"
2. "10.208.80.1"
3. "ifname"
4. "eth0"

In the asic, default route still points to the previous nexthops.

BGP router identifier 10.1.0.32, local AS number 65100
RIB entries 13010, using 1423 KiB of memory
Peers 48, using 214 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
fc00::2 4 65200 3270 3309 0 0 0 01:04:01 6402
fc00::a 4 65200 3270 3309 0 0 0 01:04:01 6402
fc00::12 4 65200 3270 3309 0 0 0 01:04:02 6402
fc00::1a 4 65200 3270 4593 0 0 0 01:04:04 6402
fc00::22 4 65200 3270 108 0 0 0 01:04:05 6402
fc00::2a 4 65200 3270 4215 0 0 0 01:04:04 6402
fc00::32 4 65200 3270 3309 0 0 0 01:04:01 6402
fc00::3a 4 65200 3270 3309 0 0 0 01:04:01 6402
fc00::42 4 64001 70 3308 0 0 0 01:04:01 6
fc00::46 4 64002 69 3307 0 0 0 01:04:00 6
fc00::4a 4 64003 70 3308 0 0 0 01:04:02 6
fc00::4e 4 64004 70 3308 0 0 0 01:04:02 6
fc00::52 4 64005 70 3308 0 0 0 01:04:02 6
fc00::56 4 64006 70 3308 0 0 0 01:04:01 6
fc00::5a 4 64007 70 4395 0 0 0 01:04:05 6
fc00::5e 4 64008 70 3308 0 0 0 01:04:01 6
fc00::62 4 64009 70 3308 0 0 0 01:04:01 6
fc00::66 4 64010 70 3308 0 0 0 01:04:01 6
fc00::6a 4 64011 70 3308 0 0 0 01:04:01 6
fc00::6e 4 64012 70 3308 0 0 0 01:04:01 6
fc00::72 4 64013 70 3308 0 0 0 01:04:01 6
fc00::76 4 64014 70 3447 0 0 0 01:04:03 6
fc00::7a 4 64015 70 3308 0 0 0 01:04:01 6
fc00::7e 4 64016 70 4375 0 0 0 01:04:05 6

But only part of the routes are synced to the kernel:
root@str-s6000-acs-11:/home/acsadmin# ip -6 route show | grep PortChannel0 | wc -l
5565
root@str-s6000-acs-11:/home/acsadmin# ip -6 route show | grep PortChannel8 | wc -l
5565
root@str-s6000-acs-11:/home/acsadmin# ip -6 route show | grep PortChannel16 | wc -l
3
root@str-s6000-acs-11:/home/acsadmin# ip -6 route show | grep PortChannel24 | wc -l
3372
root@str-s6000-acs-11:/home/acsadmin# ip -6 route show | grep PortChannel32 | wc -l
5564
root@str-s6000-acs-11:/home/acsadmin# ip -6 route show | grep PortChannel40 | wc -l
5565
root@str-s6000-acs-11:/home/acsadmin# ip -6 route show | grep PortChannel48 | wc -l
3

INFO docker/22083d1ead13(database)[726]: 20:M 14 Mar 18:15:14.068 # Client id=6731 addr=/var/run/redis/redis.sock:0 fd=68 name= age=72089 idle=2 flags=U db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=963 omem=18911864 events=rw cmd=subscribe scheduled to be closed ASAP for overcoming of output buffer limits.
INFO docker[4400]: 20:M 14 Mar 18:15:14.068 # Client id=6731 addr=/var/run/redis/redis.sock:0 fd=68 name= age=72089 idle=2 flags=U db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=963 omem=18911864 events=rw cmd=subscribe scheduled to be closed ASAP for overcoming of output buffer limits.
ERR orchagent: :- main: Exception: Unable to read redis reply

Saw something like this. Need to check.

[....] Starting enhanced syslogd: rsyslogdrsyslogd: pidfile '/var/run/rsyslogd.pid' and pid 29 already exist.
This will prevent the docker from starting instantly.

Command output:

$ sudo portstat
[Errno 2] No such file or directory 2