error when running fsort twice about ftools HOT 10 CLOSED

Can you try updating ftools and then running ftools, compile (and restarting Stata then).

Else, what version of stata/ftools are you running?

from ftools.

net install ftools, from(https://github.com/sergiocorreia/ftools/raw/master/src/)

I get the same error:

. sysuse auto, clear
(1978 Automobile Data)

. fsort turn

. fsort turn
              st_store():  3200  conformability error
                 <istmt>:     -  function returned error
r(3200);

Version:

. which ftools
/home/mauricio/ado/plus/f/ftools.ado
*! version 2.18.1 17sep2017

. which fsort
/home/mauricio/ado/plus/f/fsort.ado
*! version 2.9.0 28mar2017

This happens for any variable, but if it gives a clue the error for strings is different

. fsort make

. fsort make
AMC not found
r(111);

from ftools.

I just pushed an update that rewrites -fsort-.

It has three main differences:

• Now it actually uses Andrew Maurer's trick (discussed in code) that saves us the work of rearranging the dataset contents by hand
• It has checks for the corner case where the dataset is already sorted by varlist
• Instead of being faster above 10mm obs, it appears it is faster above 100k obs.

Note that I haven't done enough testing so this command is still in beta quality.

By the way Mauricio, I think that if you combine your plugin with Andrew Maurer's trick, you could end up with a sorting algorithm way faster than the current sort (and that fsort), which would be incredibly amazing (even if it is only faster when sorting categoricals)

from ftools.

The memory overhead is insane for sorting because I have to copy the data twice, so half the run-time is reading and writing the data. Stata uses no additional memory for sorts; I assume it's just doing quick-sort in place. I'll post a baby version of a sorting pluign later today so you can see what I mean.

from ftools.

The fsort error is gone in the latest version, btw.

from ftools.

The fsort error is gone in the latest version, btw.

Cool!

so half the run-time is reading and writing the data

With the trick that the new fsort uses, you never have to copy the data, only the variables you are sorting. Then Stata does the reshuffling within Stata based on the index you provide.

from ftools.

This fails:

. sysuse auto, clear
(1978 Automobile Data)

. gen x = "some"

. sort x make

. fsort make
some not found
r(111);

I think you need, in line 24,

		qui replace `sortvar' = `val' in 1
		cap replace `sortvar' = `"`val'"' in 1

from ftools.

This should be fixed in the new commit; I also rewrote the code to be check the types instead of capturing all the replacements.

from ftools.

(In case you are benchmarking the program, just fixed a typo created by the previous commit, that caused fsort to create missings)

from ftools.

Ok. I haven't run benchmarks yet. I've uploaded a repo with a proof-of-concept version. It's only faster for sorting on fewer variables. Sorting on entire data-sets (if there are strings) can be slower. Anyway, the repo is up if you want to test it: https://github.com/mcaceresb/stata-qsort

from ftools.