Test PCIe switches and adapters about raspberry-pi-pcie-devices HOT 53 OPEN

@Chaz6 - I'm actually going to try another switch, which might also be nice on the CM4, though it will likely cause a bit of strain on the connector... https://www.ebay.com/itm/PCIe-1x-to-Dual-PCIe-1x-90-Degree-Riser-Card-with-Power-PCIe-Switch-Expansion/273855896755?hash=item3fc3155cb3:g:fVwAAOSwHu5c4uvd

from raspberry-pi-pcie-devices.

With the 1 to 2 ports adapter, I'm able to get multiple devices to load, so that's nice:

$ lspci
00:00.0 PCI bridge: Broadcom Limited Device 2711 (rev 20)
01:00.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
02:01.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
02:02.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
03:00.0 SATA controller: Marvell Technology Group Ltd. Device 9215 (rev 11)
04:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981

I'm going to confirm I can only transfer data at up to 3.2 Gbps though. Maybe less with the overhead of the PCI bridge?

from raspberry-pi-pcie-devices.

Sorry if this is a dumb question - with using these that have a USB cable - do these perform as if they were simply USB devices or are they using a USB interface only?

They're using USB3 cables purely as a convenient physical cable. They do not convert to/from USB at an electrical or protocol level.
Connecting to a genuine USB host or slave may well result in damage to either or both ends.

As Jeff noted, there is also no standard wiring with these adapters, so ensure you use the correct pairs of adapters.

from raspberry-pi-pcie-devices.

I found another one possibly worth testing: https://www.ebay.co.uk/itm/Excellent-PCI-e-Express-1X-to-3-1X-Switch-Multiplier-HUB-Riser-Card-USB-Cable-GN/353247990975

from raspberry-pi-pcie-devices.

@Chaz6 I have that card (or a variant of it, anyway. Same design). The sticker covers it, but the chip on the board is of the same series as the IOCrest card that Jeff already tested, albeit one with a slightly expanded feature set (adding an extra port). They utilize the same kernel driver, so should ideally function exactly the same.

That said, I'm going to test it when my CM4 turns up one of these days and report back the results.

from raspberry-pi-pcie-devices.

Just to give an update, I figured out what the problem was (or is) after some digging into the switch and PCIe documentation. Not really related to the CM4, but probably useful to everyone thinking about cutting up their 1x PCIe switch ports to plug many multi-lane cards into their CM4.

When you just google about cutting open 1x PCIe ports everyone will tell you, it'll work no problem. That is not entirely true though, depending on how presence detection is implemented on the PCIe carrier board. PCIe cards have two pins, PRSNT1# and PRSNT2#, which need to be directly connected on the card. PCIe cards with more than one lane have multiple PRSNT2# pins, one for each lane (i.e. one for 1x, 4x, 8x, 16x), but only the one at the maximum number of supported lanes has to be connected to PRSNT1#.

Now, when you cut open your 1x port, PRSNT1# is connected to a pin that is not even present on the slot connector, and the 1x PRSNT2# is floating since the card supports more lanes. Thus, the PCIe switch does not detect that a card is plugged in and disables the port. When you use a proper 1x to Nx adapter, the correct PRSNTx# pins will usually be connected and the card will be detected.

Presence detection via PRSNTx# can be disabled in the switch, but the EEPROM is not fitted on the board, and you'd need to modify it's contents even if it was there. One other whacky solution, if you don't want to order a proper raiser, I used is to get out the soldering iron and botch a wire between PRSNT1# and PRSNT2# on the board, so the switch thinks there is always a card plugged in. PRSNT1# is connected to ground on this particular board, and PRSNT2# connected to the PCIe switch. I wouldn't really recommend doing this, but if you are too cheap to order yet an other PCIe adapter, it's probably fine.

And see, now cards with more than one lane are detected on the host.

from raspberry-pi-pcie-devices.

I also wanted to post an update—earlier I do remember having at least the Marvell 9215 (IO Crest) SATA card and Samsung 970 EVO NVMe working at the same time on my 2-slot switch, so it did work at least. I'll have to continue doing more testing. I have another new riser that has two slots and a different chip.

from raspberry-pi-pcie-devices.

We have seen similar idiosyncratic PCIe behaviour with the CM4 on our in-house designed carrier board for the Viota Pi-Cube and initially blamed signal integrity. The design employs an ASM1182e two port chip with downstream to a VL805 and an M.2 connector for NMVE SSDs. We found this chip (and the 4 port variant) to be the most reliable of the several bridge chip vendors that we tested on various PCIe adapter boards on the CM4IO board. The problem was a random failure to enumerate one or other or any of the downstream devices or a failure to enumerate the NVME device at 5GTs - not getting beyond 2.5GTs. We burned many hours trying to track this problem down including re-iterating our board design (to improve signal integrity) but then after one of the several CM4 eeprom firmware upgrade releases the problem seemed to disappear and enumeration of both downstream devices and at 5GTs became significantly more reliable. No other changes were made on the test rig other than the firmware upgrade so our conclusion was that whilst good enough timing at the hardware level is important, the enumeration process is also likely relevant, as in what order things get done and when. The code is closed source so it is only a best guess but IMHO worth knowing. It sounds like there may be some similar "timing" problems with the Pi5 and as mine has just arrived this morning it is high on the list to investigate. A Pi5 PCIe connector pinout would be helpful, so if anyone has worked that out it would be a big help to post it here - with the usual caveat emptor !! We plan on making some Pi5 PCIe to M.2 adapter boards and will make them available to the community when done.

from raspberry-pi-pcie-devices.

To test NVMe + SATA, or anything plus anything requiring extra power... I'm going to need to grab an external PSU. I just don't have enough adapter cables to get from my single 4-pin molex power adapter to both the floppy connector on the PCIe switch plus SATA power for the drives.

Might as well buy a PC case at some point to contain all this crazy mess haha.

Anyways, I have my eye on the Redragon GC-PS003 600W fully modular PSU... mostly because it's close, relatively cheap, and available whenever. But I'm doing plenty of other stuff so it's not yet a priority.

Edit: Just ordered that PSU, and I'll have a little more fun with the switch next week!

from raspberry-pi-pcie-devices.

The PSU is quite girthy, and also has to have a jumper across two pins on the motherboard connector so it will switch on. That was unforeseen, but obvious in hindsight.

from raspberry-pi-pcie-devices.

I'm currently testing that card from @Chaz6 in my PC (CM4 not here yet), but I'm having trouble getting multi-lane cards to work. Have you been experiencing the same? The only 1x card I have is a Realtek Gigabit card, which seems to work fine. But I haven't gotten any other card to work, I tested a Samsung 970 EVO (with 4x adapter), Mellanox 2 10GbE card and a dual gigabit card pulled from a HP server, none working. I cut open the connector to plug in longer cards since I don't have a riser card right now, but the 1x card still works in that port (so I guess nothing broke).

The cards not working do light up or get warm, but nothing in lspci, and nothing obvious in dmesg.

from raspberry-pi-pcie-devices.

@Chaz6 I have an almost identical 3 way card https://www.ebay.co.uk/itm/Excellent-PCI-e-Express-1X-to-3-1X-Switch-Multiplier-HUB-Riser-Card-USB-Cable-L-/154052637457.
Works fine with all the cards I've plugged into it so far (providing they work without the switch of course)

from raspberry-pi-pcie-devices.

is anyone able to test ASM1184e chipset, I'm planing to build my own riser card with 1 x1 to 2 x4 + 2 M.2

It uses ASMedia chip ASM1184e: https://www.aliexpress.com/item/32868079091.html
Used to be popular in mining community.

from raspberry-pi-pcie-devices.

@geerlingguy Wow dude, that is awesome what you are trying to do.

from raspberry-pi-pcie-devices.

It is best to apply the additional power. Unless the vendor claims otherwise, the additional power supply is mandatory. Then check if the PCIe switch is being enumerated. Next, of course, check if your docked widget is being found through the PCIe switch.

from raspberry-pi-pcie-devices.

@markus-k - I can't speak to @Chaz6's card, but the IO Crest seems to so far only work with devices plugged in through slot 2 (maybe slot 1 is dead, maybe something in the Pi's PCIe stack doesn't work with it)... but I haven't done any rigorous testing with it yet.

I'm still focused right now on a number of individual cards and haven't yet done thorough testing with multiple cards.

from raspberry-pi-pcie-devices.

This one seems to be working too: https://www.aliexpress.com/item/32960541989.html

from raspberry-pi-pcie-devices.

@danielkucera - Can you confirm having more than one device plugged in works, identifying all the devices? I noticed that many times it would only show devices plugged into the 2nd slot on my switch.

from raspberry-pi-pcie-devices.

Can you try one of these PCIe to PCI(-X) cards too? Like this for example.

It would make it possible to use many old PCI cards on the RPi

from raspberry-pi-pcie-devices.

@darkstar Tried already - https://www.raspberrypi.org/forums/viewtopic.php?p=1772997#p1772997
Results not positive as most cards seem to rely on mapping as I/O registers as that was the way that life used to work on x86 systems.

from raspberry-pi-pcie-devices.

I have just put two cards in the first one pictured in Jeff's post and get this:

root@raspberrypi:~# lspci
00:00.0 PCI bridge: Broadcom Limited Device 2711 (rev 20)
01:00.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
02:01.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
02:02.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
03:00.0 SATA controller: Marvell Technology Group Ltd. Device 9215 (rev 11)
04:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
04:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
root@raspberrypi:~# dmesg | grep igb
[    3.616121] igb: Intel(R) Gigabit Ethernet Network Driver
[    3.616139] igb: Copyright (c) 2007-2014 Intel Corporation.
[    3.616786] igb 0000:04:00.0: enabling device (0000 -> 0002)
[    3.842546] igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection
[    3.842572] igb 0000:04:00.0: eth1: (PCIe:2.5Gb/s:Width x1) 80:61:5f:05:0b:ae
[    3.842676] igb 0000:04:00.0: eth1: PBA No: E43709-006
[    3.842694] igb 0000:04:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[    3.843153] igb 0000:04:00.1: enabling device (0000 -> 0002)
[    4.035061] igb 0000:04:00.1: Intel(R) Gigabit Ethernet Network Connection
[    4.035087] igb 0000:04:00.1: eth2: (PCIe:2.5Gb/s:Width x1) 80:61:5f:05:0b:af
[    4.035192] igb 0000:04:00.1: eth2: PBA No: E43709-006
[    4.035208] igb 0000:04:00.1: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)

What does seem to be the case though is that it will only recognise pciex1 ie no pciex4 on a riser, I cannot get it to recognise a single pciex4 device.

from raspberry-pi-pcie-devices.

I have tried the one dlednik recommended, this is the one for us guys!!

On my Amazon it is listed as a different brand but looks like its the same as is the case with many of these devices:

https://www.amazon.co.uk/gp/product/B07NZ5W8KJ/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1

To test it, I plugged in four devices, all four worked and the cool thing is that this is a mixture of x1 and x4 devices

All four devices working. A 4x Intel 10G network card, a 1x 2 port network card, a marvel 1x 4 port sata card and a 4x nvme card

00:00.0 PCI bridge: Broadcom Limited Device 2711 (rev 20)
01:00.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
02:01.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
02:03.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
02:05.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
02:07.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
03:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
04:00.0 Non-Volatile memory controller: Sandisk Corp Device 5009 (rev 01)
05:00.0 SATA controller: Marvell Technology Group Ltd. Device 9215 (rev 11)
06:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
06:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

pi@raspberrypi:~ $ dmesg | grep ixgbe
[    3.517035] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[    3.517054] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[    3.522232] ixgbe 0000:03:00.0: enabling device (0000 -> 0002)
[    4.738177] ixgbe 0000:03:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[    4.738550] ixgbe 0000:03:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x1 link at 0000:00:00.0 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)
[    4.738912] ixgbe 0000:03:00.0: MAC: 2, PHY: 1, PBA No: G18786-004
[    4.738926] ixgbe 0000:03:00.0: 90:e2:ba:84:5e:f8
[    4.746219] ixgbe 0000:03:00.0: Intel(R) 10 Gigabit Network Connection
[    4.746761] libphy: ixgbe-mdio: probed
[    4.747202] ixgbe 0000:03:00.1: enabling device (0000 -> 0002)
[    5.932951] ixgbe 0000:03:00.1: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[    5.933290] ixgbe 0000:03:00.1: 4.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x1 link at 0000:00:00.0 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)
[    5.933631] ixgbe 0000:03:00.1: MAC: 2, PHY: 1, PBA No: G18786-004
[    5.933637] ixgbe 0000:03:00.1: 90:e2:ba:84:5e:f9
[    5.937171] ixgbe 0000:03:00.1: Intel(R) 10 Gigabit Network Connection
[    5.937435] libphy: ixgbe-mdio: probed

pi@raspberrypi:~ $ dmesg | grep igb
[    3.572119] igb: Intel(R) Gigabit Ethernet Network Driver
[    3.572136] igb: Copyright (c) 2007-2014 Intel Corporation.
[    3.572567] igb 0000:06:00.0: enabling device (0000 -> 0002)
[    3.760881] igb 0000:06:00.0: Intel(R) Gigabit Ethernet Network Connection
[    3.760906] igb 0000:06:00.0: eth1: (PCIe:2.5Gb/s:Width x1) 80:61:5f:05:0b:ae
[    3.760999] igb 0000:06:00.0: eth1: PBA No: E43709-006
[    3.761015] igb 0000:06:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[    3.761513] igb 0000:06:00.1: enabling device (0000 -> 0002)
[    3.946915] igb 0000:06:00.1: Intel(R) Gigabit Ethernet Network Connection
[    3.946940] igb 0000:06:00.1: eth2: (PCIe:2.5Gb/s:Width x1) 80:61:5f:05:0b:af
[    3.947033] igb 0000:06:00.1: eth2: PBA No: E43709-006
[    3.947049] igb 0000:06:00.1: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)

pi@raspberrypi:~ $ dmesg | grep nvme
[    3.463076] nvme nvme0: pci function 0000:04:00.0
[    3.463299] nvme 0000:04:00.0: enabling device (0000 -> 0002)
[    3.497918] nvme nvme0: allocated 32 MiB host memory buffer.
[    3.506935] nvme nvme0: 4/0/0 default/read/poll queues
[    3.516354]  nvme0n1: p1

pi@raspberrypi:~ $ dmesg | grep SATA
[    3.604608] ahci 0000:05:00.0: AHCI 0001.0000 32 slots 4 ports 6 Gbps 0xf impl SATA mode
[    3.650437] ata1: SATA max UDMA/133 abar m2048@0x601540000 port 0x601540100 irq 61
[    3.650455] ata2: SATA max UDMA/133 abar m2048@0x601540000 port 0x601540180 irq 61
[    3.650471] ata3: SATA max UDMA/133 abar m2048@0x601540000 port 0x601540200 irq 61
[    3.650486] ata4: SATA max UDMA/133 abar m2048@0x601540000 port 0x601540280 irq 61
[    4.123960] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    4.128017] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    4.128100] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    4.131946] ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)

Happy days :D

from raspberry-pi-pcie-devices.

I have just put two cards in the first one pictured in Jeff's post and get this:

[...]

What does seem to be the case though is that it will only recognise pciex1 ie no pciex4 on a riser, I cannot get it to recognise a single pciex4 device.

@PBXForums Have a look at my previous comment: #14 (comment). Check if your Riser properly connects both PRSNT# pins.

from raspberry-pi-pcie-devices.

@markus-k They work in the one in my above post, the two port one is now being scrapped in favor of that one which seems to be working perfectly.

from raspberry-pi-pcie-devices.

Some devices (like the CM4IO) don't check presence with the PRSNT# pins but whether theres a 50 Ohm load on the PCIe lanes. You can achieve the same effect on the Pericom switches by just shorting the PRSNT# pins (or configuring the switch, which is a lot harder to do).

from raspberry-pi-pcie-devices.

@markus-k Thanks, I want at least three ports anyway.

from raspberry-pi-pcie-devices.

Testing this again with the IOCrest switch but with my 700W power supply for a NAS build... I finally seem to have it powered stably enough to get two devices working, and the system doesn't completely lock up when I run lspci:

$ lspci
00:00.0 PCI bridge: Broadcom Limited Device 2711 (rev 20)
01:00.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
02:01.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
02:02.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
03:00.0 SATA controller: JMicron Technology Corp. Device 0585
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. Device 8125 (rev 04)

I was trying the same thing with the Exacq switch, and kept either having only one of the two cards work, or the Pi would lock up at various points through the boot process.

Note that I tried powering it both via dedicated 2A molex power supply and via my 700W (100+W to the 5v bus) PSU, and it seemed to work more stably connected through my PSU. It still required one extra reboot before both cards were fully functional though.

from raspberry-pi-pcie-devices.

Seems to be extremely stable now (the IOCrest switch). I think for any of these switches, a solid power supply capable of handling 3-5A or more is going to be pretty important.

But even then, some of the adapters and switches just... fail sometimes, and cause kernel panic. No idea why that happens randomly.

from raspberry-pi-pcie-devices.

Sorry if this is a dumb question - with using these that have a USB cable - do these perform as if they were simply USB devices or are they using a USB interface only?

My goal is to connect a drive and a VPU (like the coral).

In doing that, would it be better to just leave the drive on USB?

If this isn't the place to ask, feel free to delete or point me in the right direction to learn more.

from raspberry-pi-pcie-devices.

I've had my eye on these for my mining shinanigans and they look perfect for Raspberry Pi usage given their connectivity is a USB socket instead of a PCIe slot but can't seem to find them anywhere else and the vendor didn't answer my request for quote but thought I'd share anyway incase anyone knows where to get them.

https://www.alibaba.com/product-detail/pcie-splitter-pcie-to-pci-adapter_60572718142.html

from raspberry-pi-pcie-devices.

Was anyone able to test the PI7C9X2G404SL chip?
I just need a working PCIe switch that works with the CM4, Ubuntu Core 20.
Unless I have to look at the ASM1184e PCIe switch variants?

from raspberry-pi-pcie-devices.

Was anyone able to test the PI7C9X2G404SL chip?
I just need a working PCIe switch that works with the CM4, Ubuntu Core 20.

I tried that chip in Ubuntu Core 20 without success -- but I haven't confirmed the adapter card definitely works. Interested to know experience of other people. -- not had chance to look to see if it was a driver issue for ubuntu

from raspberry-pi-pcie-devices.

A PCIe compliant switch is supported by the native pci driver. Unless the hardware is defective, the PI7C9X2G404SL should be enumerated across all platforms. Full disclosure, our company is a mfr of many PCIe designs and use this component with success.

from raspberry-pi-pcie-devices.

Thanks @juthi that's what I wanted to know. Looks like poor build quality, have another one on order so will be able to confirm for @RuhanSA079 .

from raspberry-pi-pcie-devices.

Bought three of the Exacq Technologies 1 to 2 ports PCIe x1 adapter none of them show up with lspci - all broken?

from raspberry-pi-pcie-devices.

@100nix, are you powering the PCIe adapter using the molex connector? That is required.

from raspberry-pi-pcie-devices.

Thanks @juthi - yes, see picture - are there any LEDs on the card that should light up when operational? I wonder if all three cards I bought are duds.

from raspberry-pi-pcie-devices.

Personally do not have this adapter and as a company, we did not use the TI bridge ages ago due to compatibility issues with the PCIe enumeration. We camped onto the PEX8112 bridge which worked in every brand we tested - we have 1M+ adapters in the field using their bridge. Early on a major brand name of PC had enumeration faults but they corrected the issue via BIOS code within 48 hours of notification.

1. Be sure the adapter is properly seated inside your motherboard.

2. It is possible that the PCIe fanout board REQUIRES a seated PCIe adapter inside one of its PCIe slots to enable. Do you have a PCIe x1 board for testing ?

3. If possible, measure the voltages on the large coil on the adapter to confirm that the onboard DC-DC converter is working.

4. LEDs would have been nice - we use them always for such purposes but do not see it on the adapter from the posted pix.

Forgot to note, this is a standard PCIe bridge (fanout) design. You could also test the same inside of your Windows PC.

Just be sure that the PCIe gold finger keying is correct & apply the PC power supply molex connector then check your Device Manager.

If you are DOS person - you can use one of the freeware PCI utilities to scan & dump your PCIe bus to see if it enumerating on your PC.

from raspberry-pi-pcie-devices.

Thanks @juthi - plugged in one of my PCI to NVMe adapters - doesn't see it.

Here is what I see in dmesg:

[ 1.276644] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[ 1.279389] brcm-pcie fd500000.pcie: No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[ 1.282156] brcm-pcie fd500000.pcie: MEM 0x0600000000..0x07ffffffff -> 0x0000000000
[ 1.284922] brcm-pcie fd500000.pcie: IB MEM 0x0000000000..0x01ffffffff -> 0x0200000000
[ 1.606276] brcm-pcie fd500000.pcie: link down

from raspberry-pi-pcie-devices.

Hi. Not a rpi guy...yet but suggest the following:

1. if the Linux distribution offers the following, please try:

lspci -vv ; that is 2 * 'v' as in Victor and not a 'w'

this will display all PCI devices found by the host CPU. Inside this listing, check to see if you can locate the Texas Instrument PCIe switch. From the TI datasheet, you can find the PCI IDs (Vendor & Device ID). If found, then the PCIe switch is being detected by the host CPU. Then could be a s/w issue with the RPI distribution but not sure.

Respectively, you can also seat the same adapter inside a normal PC and check to see if the switch is found. These switches (like others) use the CLASS CODE to be auto-enumerated by every operating system. Respectively, no device driver is required for your Windows / Linux OS running on your x86 / AMD / Intel CPU. This is to only validate that the adapter is working.

2. I did watch the linked video at the start of this thread and the seating inside the keyed PCIe slot looks correct. Assuming that the external molex adapter is supplying enough current, you should be operational. Have you tested both of the x1 black PCIe slots for your NVMe adapter ? Can only assume that the vendor shipped you tested boards.

3. If you remove this adapter and seat in your PCIe to NVMe adapter -> does the rpi see the NVMe adapter natively (without this PCIe switch)? That will be a good test. I am assuming that your NVMe adapter is for the PCIe slot and not the PCI slot otherwise that adds another layer of complication (rpi PCIe slot -> PCIe switch (TI) -> PCIe to PCI bridge (?? brand / make) -> PCI NVMe adapter. It can be done but with care. We have built many designs like this early on for our product line.

from raspberry-pi-pcie-devices.

Silly but important concern, can you post a clear picture of the molex connector onboard your external power adapter? Also check against the industry standard pinout. A standard PCIe slot offers 3v3 and 12v0 rails. Be sure this molex connector is supplying these voltages to the proper PCIe adapter pins.

from raspberry-pi-pcie-devices.

Hey @juthi

I moved on for the moment to play around with the ASHATA PCE4PCE-A01 1 to 4 Port PCIe Switch.

It seems to work, as you can see on the picture I am using the same power adapter with a sata power plug.

I will revert back once I get a chance to try the Exacq Technologies 1 to 2 ports PCIe x1 adapter again.

Thanks

Alex

from raspberry-pi-pcie-devices.

Thanks @juthi - yes, see picture - are there any LEDs on the card that should light up when operational? I wonder if all three cards I bought are duds.

I'm not sure if it's the issue here, but try powering the pi and the card from one power supply, e.g. a PC power supply, as there have been issues with using multiple power supplies (bad ground connection resulting in connection issues. Jeff had those issues with GPUs)

from raspberry-pi-pcie-devices.

Excellent advice @Coreforge. On a standard PC PCIe slot, the PCIe adapter must be fully powered and awake before the host bios can enumerate the PCIe function. On some protos, we intentionally had some soft start power rails to reduce in-rush currents and found that some branded PCs would miss the detection of our adapter. This may not be the same issue with the rpi but the idea to test with a PC power supply is a good one. Furthermore, the CM4 is powered from a 12V power adapter (coax / barrel jack). You could use a common PC power supply to power the molex connector on the PCIe switch + apply the yellow (+12v) and black (ground) from the same PC power supply onto the coax connector. Be sure to check on the polarity of the coax connector and their through hole location on the CM4 PCB. Do not reverse wire the power supply!!

A few more comments - the TI PCIe switch is a Gen1 (2.5Gbps) switch for the fanout. The CM4 is reported to be a Gen2 PCIe switch (5.0Gbps). If the 5.0 Gbps is confirmed then it will be more practical to install a Gen2 PCIe switch to allow for the downstream adapters to run faster than with the TI switch.

In the meantime, can you share the output from the following (found from Jeff's logs):

lspci -vvnnk ; run this command on your CM4
00:00.0 PCI bridge [0604]: Broadcom Limited Device [14e4:2711] (rev 20) (prog-if 00 [Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 64
Bus: primary=00, secondary=01, subordinate=02, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: c0000000-c00fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: access denied
Kernel driver in use: pcieport

01:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 01) (prog-if 00 [Normal decode])
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 64
Bus: primary=01, secondary=02, subordinate=02, sec-latency=0
Memory behind bridge: c0000000-c00fffff
Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort+ >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: access denied

02:00.0 FireWire (IEEE 1394) [0c00]: VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller [1106:3044] (rev 80) (prog-if 10 [OHCI])
Subsystem: VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller [1106:3044]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping+ SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 0
Region 0: Memory at 600000000 (32-bit, non-prefetchable) [disabled] [size=2K]
Region 1: I/O ports at [disabled]
Capabilities: access denied

The above should list all enumerated devices on the CM4 and should also display the PCIe switch if it is being enumerated.

reference:
https://gist.github.com/geerlingguy/9d78ea34cab8e18d71ee5954417429df

from raspberry-pi-pcie-devices.

The IO Board also has a floppy connector for power that can just be used with an adapter and an atx power supply, or just an older one that still has a floppy connector. Especially with older ones, but maybe also with newer ones, they need some kind of base load though on either the 5V or 12V rail to have a stable output and some just shut off without one. Either a power resistor or a turn indicator bulb work well as one.

from raspberry-pi-pcie-devices.

Hello. Do you always need to connect additional power to the board? Or only when the device needs more power?

from raspberry-pi-pcie-devices.

I know I'm kind of late to the party, but what version of raspbian did you do your initial testing on ? My freshly installed Rasbian 11 32-bit just just outputs segmentation fault when running lspci as a regular user, and ust freezes when same command run as root.

the board I'm using is the Exacq Technologies 1 to 2 ports PCIe x1 adapter using the TI XIO3130 PCIe switch chip powered through the J20 powerheader.

Update: Running now on rasbian 11 64-bir and I have now been able to get the PCIe switch recognized by the OS. I have tried to plug both a NVMe adapter and a Realtek Nic into it and only the NIC is recognized, The driver for the nic is loaded but no device is present in the OS, just eth0. Could this be a power issue as it is not stable. Sometimes it just freezes when issuing lspci

from raspberry-pi-pcie-devices.

Testing with the Raspberry Pi 5 and an NVMe drive...

pi@pi5:~ $ lspci
0000:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21)
0000:01:00.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
0000:02:01.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
0000:02:02.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
0001:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21)
0001:01:00.0 Ethernet controller: Device 1de4:0001

For some reason I'm not seeing the NVMe drive. Going to try something else.

from raspberry-pi-pcie-devices.

My soundblaster card shows up behind the bridge, though:

pi@pi5:~ $ lspci
0000:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21)
0000:01:00.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
0000:02:01.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
0000:02:02.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
0000:04:00.0 Audio device: Creative Labs EMU20k2 [Sound Blaster X-Fi Titanium Series] (rev 03)
0001:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21)
0001:01:00.0 Ethernet controller: Device 1de4:0001
pi@pi5:~ $ lspci -tv
-+-[0000:00]---00.0-[01-04]----00.0-[02-04]--+-01.0-[03]--
 |                                           \-02.0-[04]----00.0  Creative Labs EMU20k2 [Sound Blaster X-Fi Titanium Series]
 \-[0001:00]---00.0-[01]----00.0  Device 1de4:0001

Hmm... maybe it's just one of the two ports on my riser is having an issue, because swapping the NVMe to the other side gets it to show up:

pi@pi5:~ $ lspci
0000:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21)
0000:01:00.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
0000:02:01.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
0000:02:02.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
0000:04:00.0 Non-Volatile memory controller: KIOXIA Corporation Device 0010 (rev 01)
0001:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21)
0001:01:00.0 Ethernet controller: Device 1de4:0001
pi@pi5:~ $ lspci -tv
-+-[0000:00]---00.0-[01-04]----00.0-[02-04]--+-01.0-[03]--
 |                                           \-02.0-[04]----00.0  KIOXIA Corporation Device 0010
 \-[0001:00]---00.0-[01]----00.0  Device 1de4:0001

Edit: Hmm, now it's showing fine in slot 1 too, lol. Maybe just bad luck!

pi@pi5:~ $ lspci -tv
-+-[0000:00]---00.0-[01-04]----00.0-[02-04]--+-01.0-[03]----00.0  KIOXIA Corporation Device 0010
 |                                           \-02.0-[04]--
 \-[0001:00]---00.0-[01]----00.0  Device 1de4:0001

from raspberry-pi-pcie-devices.

Seems stable now, but like on the CM4, it looks like the firmware only looks for root-level NVMe drives—at least right now. I'll ask RPi about whether they'll support deeper NVMe enumeration for boot.

from raspberry-pi-pcie-devices.

In case anybody's using one of these Pericom PI7C9X2G304 / PI7C9X2G404 PCIe switches in production and experiencing random kernel panics, there is a fix in the works over here (which I have personally validated on 5.15.92):
raspberrypi/linux#5352

from raspberry-pi-pcie-devices.

@theodric - Wow, excellent news! And great sleuthing in that thread. I'm pretty sure all the issues I've seen on the Pi 5's bus have been due to clock problems with signal integrity, but now I'm also wondering if certain weird behaviors on the CM4 with other devices could be related too.

from raspberry-pi-pcie-devices.

Dr George Smart has posted his reverse engineering hack of the Pi5 PCIe connector pinout on Github and YouTube - and saved me cannibalising my Pi5! Big thank you George - will take it as valuable input for an M.2 HAT and keep you posted.

m1geo/Pi5_PCIe

from raspberry-pi-pcie-devices.