ofiwg / fabtests Goto Github PK
View Code? Open in Web Editor NEWFROZEN: the master branch has merged with the libfabric git repo
Home Page: http://libfabric.org
License: Other
FROZEN: the master branch has merged with the libfabric git repo
Home Page: http://libfabric.org
License: Other
Because injected sends do not generate a completion, the last inject call may be aborted when the app exits. The result is that the peer side hangs. The test needs to be modified to synchronize with the peer before exiting.
Backtrace looks like this:
#0 0x00007ffff6b948ec in free () from /lib64/libc.so.6
#1 0x00007ffff7b7bf62 in fi_freeinfo (info=0x605040) at src/fabric.c:276
#2 0x0000000000400fb7 in main (argc=, argv=) at simple/rdm_rma_simple.c:355
I had some printf's added so line numbers may be off by a line or two. The issue appears to be with the hints info object. IT is first allocated by fi_allocinfo. If -f is specificed, hints->fabric_attr->prov_name = optarg (argv). Then at the end of the test, fi_freeinfo(hints) is called, which in turn calls free() on hints->fabric_attr->prov_name.
Reproduce by running ./fi_rdm_rma_simple -f sockets (or any specific provider). Seems like a simple fix is to use strdup().
Several other tests may also have the same issue:
$ grep prov_name *
dgram.c:332: hints->fabric_attr->prov_name = optarg;
msg.c:465: hints->fabric_attr->prov_name = optarg;
rdm.c:329: hints->fabric_attr->prov_name = optarg;
rdm_rma_simple.c:336: hints->fabric_attr->prov_name = strdup(optarg);
This is related to commit ofiwg/libfabric@33511f05.
At the moment only two tests have a -f option for giving a fabric hint.
Does there need to be a config directory?
cst-fs:~/sfi/fabtests$ ./autogen.sh
The endpoint attribute structure should be expanded to expose the size of the underlying queue. Now that the EP attribute exist, we can simplify things for the user and avoid needing to use control interfaces to override the default values. But default values should still be available to the user, with the actual values returned when an endpoint is created.
I'm using a HEAD of master libfabric and same for fabtests. I'm getting this compile error when trying to build fabtests:
CC simple/rdm_tagged_search.o
simple/rdm_tagged_search.c: In function ‘main’:
simple/rdm_tagged_search.c:519:2: warning: implicit declaration of function ‘fi_allocinfo’ [-Wimplicit-function-declaration]
hints = fi_allocinfo();
^
simple/rdm_tagged_search.c:519:8: warning: assignment makes pointer from integer without a cast [enabled by default]
hints = fi_allocinfo();
^
CCLD simple/fi_rdm_tagged_search
simple/rdm_tagged_search.o: In function main': /home/hpp/fabtests/simple/rdm_tagged_search.c:519: undefined reference to
fi_allocinfo'
collect2: error: ld returned 1 exit status
Need ping-pong type example for using RMA. Candidate example for using counters to check for completion of RMA.
A waitset(fid_wait) can be attached to multiple event queues but fi_wait doesn't return any information about the signaled waitset. How do we know which completion queue the signaled waitset refers to?
Also note, fi_poll returns contexts associated with event queue for the corresponding pollset but it might be more generic to return fids.
Need a simple, but complete example that demonstrates using all of the atomic operations, including write coherent support.
sync, async, all flavors of av_insert
For recv_counter, we read the current counter and wait until it reaches current value plus one.
But, the counter might have been updated already.
ret = fi_cntr_wait(rcntr, fi_cntr_read(rcntr) + 1, CNTR_TIMEOUT);
Just like send counter, we need to use a separate variable to keep track of counter values.
Segfault is in memcpy from line 331 because fi->dest_addr is null after fi_getinfo call. Not sure whether this should be handled in the test or in the PSM provider.
@jithinjosepkl
Tests were merged upstream, but they fail:
name bytes xfers iters total time Gb/sec usec/xfer
Event queue read -259 (Error available)
-s argument for many examples seem to ignore the port information.
eg: in fi_rdm_pingpong/tagged, getaddrinfo is called for "-s", but without the port (even when port is specified).
cmatose uses port# 7471, which is used by Apple QuickTime Streaming Server.
This causes the test to fail on Mac.
Is it OK to change the port to something else?
Just double checking, since it is a ported example.
Tagged ping-pong example over RDM
The UD pingpong test has at least 2 issues. The first is relatively easy to fix. The av_attr structure is not fully initialized before being used. A memset 0 would work.
The second is that the server side never enters the client address into its AV. It simply expects the CQ readfrom call to have inserted the value. As a generic example for UD, I would recommend having the client call getname(), insert the results in a message, and send that to the server. The server would extract the name from the message, insert it into its AV, and send an ack response. This would be done as part of the communication setup before real data transfers actually start.
Copying @rfaucett (author) and @luomiao (who found the issues). Reese, can I assign you to this?
We are trying to run some of the FI_EP_RDM tests in fabtests/simple on our qlogic cluster.
The tests invariably fail in a getaddrinfo call within psmx_resolve_hostname with messages
like:
bash-4.1$ ./fi_rdm -d 10.16.9.48
psmx_resolve_name: couldn't connect to 10.16.9.48:4096
When we build libfabric on the intel/qlogic systems, we get socket/verbs/psm providers,
and fi_getinfo test returns sensible information. However, whenever we try to run any of the
client/server tests that use FI_EP_RDM, we hit this hostname resolution error in psmx_resolve_name -
aka getaddrinfo.
Note these are production psm-based systems, so there is not likely to be anything wrong with the base OFED setup.
Any samples that have an AV must close the associated EPs first
I can work on this
From the OFIWG F2F, there was a request to allow an application to indicate that a new connection request would be handed off to another process (forked or otherwise). The idea is that the provider could arrange its data structures accordingly, so that the new connection could successfully be migrated to another process.
The latency performance for fi_msg_rma is reporting sub half micro-second latencies. This seems low, which makes me suspect that the way the performance is being captured is incorrect.
Tagging @a-ilango
msg_epoll.c requires epoll.h:
simple/msg_epoll.c:37:10: fatal error: 'sys/epoll.h' file not found
Should there be a configure check for this?
fi_rdm_pingpong
simple/rdm_pingpong.c:353: EP opened on fabric IP
Some of the tests support a -f option to allow giving a fabric hint (such as rdm_pingpong.c). I am trying to enhance msg_pingpong.c to have a -f option and am encountering an issue. The changes I've made can be viewed in commit: 36ff792.
If I run fi_msg_pingpong as a server I get the result:
[wf143][99.35] :: export SFI_PSM_NAME_SERVER=1
[wf143][99.51] :: ./fi_msg_pingpong -f IP
EP opened on fabric IP
Which seems okay, but the client errors in this fashion:
[wf144][99.35] :: export SFI_PSM_NAME_SERVER=1
[wf144][99.35] :: ./fi_msg_pingpong -d wf143 -f IP
psmx_resolve_name: couldn't connect to wf143:4096
[SOCK_ERROR - sock_eq_openwait:430]: no available AF_INET address
no available AF_INET address: Success
[SOCK_ERROR - sock_util_sendto:71]: sendto failed with error 9 - Bad file descriptor
fi_connect Bad file descriptor
If I enable socket log output:
[wf144][99.35] :: export OFI_SOCK_LOG_LEVEL=4
[wf144][99.35] :: ./fi_msg_pingpong -d wf143 -f IP psmx_resolve_name: couldn't connect to wf143:4096
[SOCK_INFO - sock_msg_getinfo:357]: dest_addr: family: 2, IP is 192.168.0.143
[SOCK_INFO - sock_msg_getinfo:363]: src_addr: family: 2, IP is 192.168.0.144
[SOCK_INFO - sock_pe_init_table:2097]: PE table init: OK
[SOCK_INFO - sock_pe_init:2121]: PE init: OK
[SOCK_INFO - sock_pe_progress_thread:2044]: Progress thread started
[SOCK_INFO - _sock_conn_listen:274]: Binding listener thread to port: 60308
[SOCK_INFO - sock_eq_openwait:409]: enter
[SOCK_ERROR - sock_eq_openwait:430]: no available AF_INET address
no available AF_INET address: Success
[SOCK_INFO - sock_pe_add_tx_ctx:1867]: TX ctx added to PE
[SOCK_INFO - sock_pe_add_rx_ctx:1875]: RX ctx added to PE
[SOCK_INFO - sock_rx_new_entry:58]: New rx_entry: 0x985a30, ctx: 0x9232e0
[SOCK_INFO - sock_ep_recvmsg:108]: New rx_entry: 0x985a30 (ctx: 0x9232e0)
[SOCK_ERROR - sock_util_sendto:71]: sendto failed with error 9 - Bad file descriptor
fi_connect Bad file descriptor
[SOCK_INFO - sock_pe_progress_thread:2078]: Progress thread terminated
[SOCK_INFO - sock_pe_finalize:2145]: Progress engine finalize: OK
If I break the server at sock_eq_openwait I get the following:
Breakpoint 1, sock_eq_openwait (eq=0x605220, service=0x6051cc "9228") at prov/sockets/src/sock_eq.c:408
Notice that the value for service is "9228" which is correct.
But if I break the client at sock_eq_openwait I get the following:
Breakpoint 1, sock_eq_openwait (eq=0x67c8f0, service=0x6055d4 "-\243") at prov/sockets/src/sock_eq.c:408
The value of service changes everytime. Not exactly sure what is going on there.
Another issue is that the test continues after the first failure in fi_ep_bind in bind_ep_res (msg_pingpong.c:255) since ret doesn't contain an error value.
Am I missing something, or is it just not behaving as expected?
EDIT:
A bit more information:
I just re-compiled libfabric with:
./configure --prefix=/users/bturrubiates --disable-verbs
make install
I then switched to the master branch of fabtests that contains the unmodified test and attempted to execute it and I get the exact same behavior.
Is this an issue with the test, or in libfabric?
I think that the access parameter to fi_mr_reg
in fi_msg_pingpong
is incorrect (should be something like FI_SEND|FI_RECV
, not 0), and I suspect it's incorrect elsewhere in the fabtests codebase. I don't have spare cycles right now to audit things, so I'm filing this as a reminder that we need to check this at some point.
Modify ft_finalize call for rdm_shared_ctx and scalable_ep. Currently our ft_finalize is called with ep[0]. We need to add a local ft_finalize for these because each ep is individually addressable. We need to get the completion for the right ep.
ft_finalize(ep[0], scq, rcq, remote_fi_addr[0]);
The basic tests don't have src_addr option. Adding src_addr to following tests would help to unify the code structure-
From the OFIWG F2F, data structure versions will be indicated using a version parameter to fi_getinfo. The version parameter will indicate the version of the set of data structures known to the application. libfabric will adjust its behavior accordingly, based on the data structures and fields known to the app. This mechanism will replace the field/mask concept in the current data structure scheme.
The source code should reference a dual GPLv2 / BSD license.
Some of the apps take the -d dest_addr option, while others take the dest_addr on the command line by itself. The apps should be consistent in this.
@pmmccorm - Is this a change that you're working on?
Any reason rdm_pingpong does not support registering memory ?
CC: @shantonu
The master branch is currently broken with for usNIC:
$ FI_LOG_LEVEL=3 ./simple/fi_ud_pingpong
libfabric:fi_register_provider():108<2> registering provider: usnic (1.0)
libfabric:fi_register_provider():108<2> registering provider: verbs (1.0)
libfabric:fi_register_provider():108<2> registering provider: sockets (1.0)
libfabric:verbs:fi_ibv_check_info():391<2> Required mode bits not set
libfabric:fi_getinfo():432<1> fi_getinfo: provider verbs returned -61 (No data available)
libfabric:sockets:sock_ep_getinfo():324<3> src_addr: 127.0.0.1
libfabric:usnic:usdf_fabric_open():979<3> successfully opened usnic_0/eth6
Unable to resolve remote address 0x0 0x0
fi_close(): 191, -16 (Device or resource busy)
fi_close(): 197, -16 (Device or resource busy)
I haven't tracked down the root cause, but PR #194 seems to be the culprit, specifically commit f255f0d. I haven't been able to spot the bug yet by inspection, so I'll start proper debugging soon.
fi_rdm_pingpong
is also failing, though with different symptoms:
$ FI_LOG_LEVEL=3 FI_PROVIDER=usnic ./simple/fi_rdm_pingpong
[... start other side in other window ...]
libfabric:fi_register_provider():108<2> registering provider: usnic (1.0)
libfabric:fi_register_provider():108<2> registering provider: verbs (1.0)
libfabric:fi_register_provider():125<2> "verbs" filtered by provider include/exclude list, skipping
libfabric:fi_register_provider():108<2> registering provider: sockets (1.0)
libfabric:fi_register_provider():125<2> "sockets" filtered by provider include/exclude list, skipping
libfabric:usnic:usdf_fabric_open():979<3> successfully opened usnic_0/eth6
fi_recv(): 117, -11 (Resource temporarily unavailable)
The negative tests in AV unit-test reads a bad IP address from user and does a getaddrinfo prior to av_insert(). The getaddrinfo call fails (for an invalid IP address) and the test is reported as failed.
So, was the intended meaning of bad IP address "a valid IP address but not reachable"?
AFAIK, the av_insert() need not check for reachability.
Or is it just an issue with test itself?
There should be at least one example that uses asynchronous memory registration. Maybe this is a modification or enhancement to an RMA example.
We should make use of wait_for_completion and wait_for_data_completion in common/shared.c instead of repeating the code block in all the tests. I suggest:
int wait_for_tagged_completion(struct fid_cq *cq, int num_completions)
When the writedata test option is selected, the tests should request support for remote CQ data by setting domain_attr::cq_data_size > 0. This will allow them to detect providers that cannot support remote CQ data and fail gracefully.
fi_av_insert returns the number of successful inserts but it doesn't return negative value when it fails to insert an address since it treated as "not being able to resolve an user supplied address". In our simple examples, we return what fi_av_insert returns but in case it returns 0(success), the calling function should return failure instead of success(0).
original code:
ret = fi_av_insert(av, remote_addr, 1, &remote_fi_addr, 0, &fi_ctx_av);
if (ret != 1) {
FT_PRINTERR("fi_av_insert", ret);
return ret;
}
suggested code:
ret = fi_av_insert(av, remote_addr, 1, &remote_fi_addr, 0, &fi_ctx_av);
if(ret == 0) {
FT_DEBUG("fi_av_insert(): cannot insert address 0x%" PRIx64 "\n", (uint64_t *)remote_addr);
return -1;
} else if(ret !=1) {
FT_PRINTERR("fi_av_insert", ret);
return ret;
}
@pmmccorm You have some "cleanups for fabtests" PRs recently, so I thought I'd throw this one your way... :-)
With gcc 4.9.1:
common/shared.c: In function ‘init_test’:
common/shared.c:150:29: warning: argument to ‘sizeof’ in ‘snprintf’ call is the same expression as the destination; did you mean to provide an explicit length? [-Wsizeof-pointer-memaccess]
snprintf(test_name, sizeof test_name, "%s_lat", sstr);
^
The unit test fi_eq_test
makes use of CLOCK_MONOTONIC_RAW. Though it's been available in linux for a while, slightly less older versions of glibc did not make it available (like on some of the enterprise systems I'm using). Seems like something that could be checked at config time. FWIW, I think you could #define it by hand with the value from linux/time.h and things would work, though be a bit unsatisfying.
It doesn't matter whether or not a test run is successful or not, I still get a psmx_resolve_name error.
[wf541][85.25] :: ./fi_dgram -d wf540
psmx_resolve_name: couldn't connect to wf540:4096
Posting a send...
Send completion received
From OFIWG F2F, interface structure versioning will be done using a size field within the struct. A query method (or static inline or define) will indicate if a specific interface is available.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.