GetLLM crashes with NBcpl == 1 because fwqwf[2] is not defined. See email conversation below.

I tried it but it seems not to work.
The program crashes here:

        #-- Convert to C-matrix:
        if with_ac_calc:
            try:   
                [coupleterms,Qminav,Qminerr,bpms] = getCandGammaQmin(fwqwf,bpmsf,Q1f,Q2f,MADTwiss)
            except:
                [coupleterms,Qminav,Qminerr,bpms] = getCandGammaQmin(fwqwf2,bpmsf2,Q1f,Q2f,MADTwiss)
        else:
            [coupleterms,Qminav,Qminerr,bpms] = getCandGammaQmin(fwqw,bpms,Q1f,Q2f,MADTwiss)

fwqwf2 is not defined and an exception is raised. fwqwf and fwqwf2 will only be defined if NBcpl == 2.
I tried to fix it by defining fwqwf and fwqwf2, initializing them with None and check it with

        #-- Convert to C-matrix:
        if with_ac_calc and (fwqwf is not None or fwqwf2 is not None):
            ...
        else:
            [coupleterms,Qminav,Qminerr,bpms] = getCandGammaQmin(fwqw,bpms,Q1f,Q2f,MADTwiss)

but then occurs an exception(list index out of range) within getCandGammaQmin(fwqw,bpms,Q1f,Q2f,MADTwiss).
Here is the stack trace:

Traceback (most recent call last):
  File "C:\eclipse_4_2_2_python\workspace\externalProjects\test\getllm_code_coverage.py", line 111, in <module>
    run_getllm()
  File "C:\eclipse_4_2_2_python\workspace\externalProjects\test\getllm_code_coverage.py", line 42, in run_getllm
    higher_order=my_higher_order)
  File "C:\eclipse_4_2_2_python\workspace\externalProjects\test\GetLLM_dev.py", line 4980, in main
    [coupleterms,Qminav,Qminerr,bpms] = getCandGammaQmin(fwqw,bpms,Q1f,Q2f,MADTwiss)
  File "C:\eclipse_4_2_2_python\workspace\externalProjects\test\GetLLM_dev.py", line 2602, in getCandGammaQmin
    detC=1-(1/(1+4*(abs(fqwq[bpmm][0][0])**2-abs(fqwq[bpmm][0][2])**2)))
IndexError: list index out of range

For now I will open an issue on GitHub.

Cheers,
Viktor

On 02/05/2013 10:36, Yngve Inntjore Levinsen wrote:

Hi Viktor,

The GetCoupling1 function can be called using the argument '--nbcpl=1' I believe. It is not used much in our analysis as far as I am aware.

Cheers,
Yngve

On 02. mai 2013 10:31, Viktor Maier wrote:

Hi Yngve,

I splitted the files in three testcases, but now I created a new testcase with all three source files from getsuper test together.
We have a code coverage of 82.1%(before 72.5%) now.

Here are the still uncovered funstions:

<func_name> [<called_by_func>[<reason>] ]

Not covered:       
    GetCoupling1                                 main        (if   NBcpl==1:) and (if woliny2!=1 and wolinx2!=1:) are false
    (GetCoupling2                                main        commented out, deleted and GetCoupling2b was renamed to GetCoupling2)
    Getoctopole                                    main         commented out
    PseudoDoublePlaneMonitors       main        (if (accel=="SPS" or "RHIC" in accel) and wolinx!=1 and woliny!=1 :) is false
    (_write_llm_tfs_header        not called    deleted)
    f2h                                                    Getoctopole Getoctopole not called
    (function                                            not called    deleted)
    union                                                not called

Covered by run with three sourcefiles and dpp != 0.0 from May 2nd 2013:
    ConstructOffMomentumModel    main        (if wolinx2!=1:) and (if woliny2!=1:) are false
    DispersionfromOrbit                        main        (if wolinx!=1 and wolinx2!=1:) and (if woliny!=1 and woliny2!=1:) are false
    NormDispX                                        main        (if wolinx!=1 and wolinx2!=1:) is false
    GetDPX                                                NormDispX, main     NormDispX not executed and(if wolinx!=1 and wolinx2!=1:) is false
    GetDPY                                                main        (if woliny!=1 and woliny2!=1:) is false

Cheers,
Viktor

On 02/05/2013 09:31, Yngve Inntjore Levinsen wrote:

Hi Viktor,

You already have that, it is in the getsuper test which you took the
files from. The three files are for dpp= 0, -something and +something.

In any case I think in GetLLM it is not detected automatically, so you
can just use any file and state in the input arguments the dpp you want.

Cheers,
Yngve

On 02. mai 2013 08:36, Viktor Maier wrote:

Hi guys,

can someone of you provide me test data for getDx.out, getNDx.out and
getDy.out?
As far as I get it from the code I need several source files with
linx/liny dpp != 0.0 .

Thank you!

Cheers,
Viktor

I noticed that getkick.out and getkick_ac.out have descriptor names with whitespaces:

@ RescalingFactor for X %le 1.01054725544

The name is "RescalingFactor for X". It includes whitespaces. I'm not sure if this is allowed(SL-CO-Note-91-32 is not very detailled concerning the syntax of tfs files) but metaclass will deliver wrong results since it uses str.split() and accesses by index operator:

split_line = line.split()
if(... "s" not in split_line[2]):

Therefore I inserted underscores: "RescalingFactor_for_X"

@rogeliotomas @Eothred @alangner

I changed the behavior of the creation of GetLLM files. Before the files have always been created even if there are only descriptors in it. Now I do not create a file if the table is empty.

Especially the files getNDx.out, getDx.out and getDy.out will be omitted if they are without content.

The reason is MODEL.LHCB.model.Corrections.getdiff.py crashes because of that. It checks if these files exist. If they exist it try to get the column 'NAME' which is not available and the script crashes. In case of not existing files merely the section would be skipped.

So it is better to have no file than an empty file in this case.

@rogeliotomas @Eothred @alangner

I want to rename GetLLM since the package/dir name is the same. This is not forbidden but it produces trouble in import statements.

It is recommended to use absolute imports instead of relative imports.
Relative imports are only possible within a subdir/subpackage of a project. For example I could import in GetLLM:

    relative: import algorithms.helper
    absolute: import GetLLM.algorithms.helper

The absolute import is not possible in this case, though. The interpreter will find first the module GetLLM.py and not the package GetLLM. Since GetLLM.py has no algorithms.helper an exception will be raised.
Further it is not possible to use something from GetLLM.py in subpackages. I cannot import DEBUG from GetLLM.py in Getllm.algorithms.helper.py. Relative import is not possible since I'm in a subpackage and absolute import raises an error.

I would only need to change Programversions.properties, getsuper.py and filecheck.py.

GetPhases will be invoked wrong four times. The parameter Q is missing:

[phasexp, Q1, MUX, bpmsx] = GetPhases(getllm_data, MADTwiss, PseudoListX, 'H')

Here is the definition of GetPhases:

def GetPhases(getllm_data, MADTwiss, ListOfFiles, Q, plane):
    ....

This will only happen if accel == "SPS" or "RHIC" in accel

@rogeliotomas @Eothred @alangner

It seems that I need to create the correct model to run WAnalyses in the GUI.
I fixed the issue #15 but other errors occur which depend on the input files.
Here is a ZeroDivisionError:

17:42:59 - WAnalysis STDERR: Traceback (most recent call last):
17:42:59 - WAnalysis STDERR:   File "/afs/cern.ch/eng/sl/lintrack/Beta-Beat.src/GetLLM/getsuper.py", line 753, in 
17:42:59 - WAnalysis STDERR:     main(options,args)
17:42:59 - WAnalysis STDERR:   File "/afs/cern.ch/eng/sl/lintrack/Beta-Beat.src/GetLLM/getsuper.py", line 684, in main
17:42:59 - WAnalysis STDERR:     dolinregbet(fileobj,fileslist.keys(),betalistx,bpms,"H",zerobx,modeld)
17:42:59 - WAnalysis STDERR:   File "/afs/cern.ch/eng/sl/lintrack/Beta-Beat.src/GetLLM/getsuper.py", line 441, in dolinregbet
17:42:59 - WAnalysis STDERR:     bfit=linreg.linreg(listx, b)
17:42:59 - WAnalysis STDERR:   File "/afs/cern.ch/eng/sl/lintrack/Python_Classes4MAD/linreg.py", line 33, in linreg
17:42:59 - WAnalysis STDERR:     Var_a, Var_b = ss * N / det, ss * Sxx / det
17:42:59 - WAnalysis STDERR: ZeroDivisionError: float division

I used Yngves files from the getsuper test and a model which I created in the GUI. I only checked the AC box. All other settings I did not touch.

Do I need special settings for the model to run WAnalyses?

There are doubts about the correctness of GetLLM.Getsextupole() and therefore it should be tested.

@rogeliotomas @Eothred
I am working currently on correct_ChromCouple.py. It is pretty similar to correct_coupleDy.py. To the latter one the feature "automatic model cut" was added(see bottom).

I was wondering if this feature would also be an enhancement for correct_ChromCoup.py?

def _calculate_automatic_model_cut_if_desired(couple_twiss):
    if not"coupling_knobs" in _InputData.variables_list: # then it is not desired(vimaier)
        return _InputData.model_cut_c
    if 0 == couple_twiss.F1001W.size:
        return _InputData.model_cut_c
    print "coupling_knobs mode. Trying to do automatic correcting"
    # we want to have a value which indicates the worst 5 percent, so we sort and get the value from the 95% index (tbach)
    sortedF1001W = numpy.sort(couple_twiss.F1001W)
    five_percent_index = int(numpy.floor(sortedF1001W.size * 0.95)) # no requirement to be accurate (tbach)
    five_percent_value = sortedF1001W[five_percent_index]

    if PRINT_DEBUG:
        print "five_percent_index:", five_percent_index

    if numpy.allclose(five_percent_value, 0):
        print >> sys.stderr, "calculated value for modelcut is 0, exit"
        sys.exit(1)

    print "modelcutC will be changed from:", _InputData.model_cut_c, "to:", five_percent_value
    return five_percent_value

@rogeliotomas @Eothred @alangner

Since I inserted traceback.print_exc() in all except blocks I saw a lot of exceptions in a run with NBcpl==1. Thus I found a bug which is very tricky.
The bug is in the following line(algorithms.coupling.py:75): (Who can find the bug?)

list_row_entries = [..., mad_twiss.f1001[mad_twiss.indx(bn1)].real, mad_twiss.f1001[mad_twiss.indx(bn1)].imag, mad_ac.f1010[mad_ac.indx(bn1)].real, mad_ac.f1010[mad_ac.indx(bn1)].imag]

Error-msg: TypeError: 'dict' object is not callable

Did you find the error?

Instead of square brackets there are curved brackets:

 mad_twiss.f1001[mad_twiss.indx[bn1]]

This is also a very good example why you should never use try/except(everything).

Implementation/populating of post-processing
scripts (~30 lines) to gather tune and amplitude from Drive
and GetLLM outputs.

Rogelio:

Simon developed very nice post-processing
scripts (~30 lines) to gather tune and amplitude from Drive
and GetLLM outputs. It would be very nice to
have this in the GUI for 2014-15. We may discuss on how
to implement it, options might be:
-really in Java
-upgrade GetLLM (Simon does not see it)
-standalone, python or shell script 

nothing @rogeliotomas @Eothred

I tracked down an error from the GUI.
The reason for the following exception is that there is no BPM next to AC dipole (BPMYA.5L4.B1 or BPMYB.6L4.B19) in lin_y file and GetLLM.GetFreeBetaFromAmp_Eq() returns a default value ([{},'',[],[]]). By accessing a value an IndexError is raised.

    Traceback (most recent call last):
      File "C:\eclipse_4_2_2_python\plugins\org.python.pydev_2.7.3.2013031601\pysrc\pydevd.py", line 1397, in 
        debugger.run(setup['file'], None, None)
      File "C:\eclipse_4_2_2_python\plugins\org.python.pydev_2.7.3.2013031601\pysrc\pydevd.py", line 1090, in run
        pydev_imports.execfile(file, globals, locals) #execute the script
      File "C:\eclipse_4_2_2_python\workspace\Beta-Beat.src\GetLLM\GetLLM.py", line 5328, in 
        higher_order=options.higher)
      File "C:\eclipse_4_2_2_python\workspace\Beta-Beat.src\GetLLM\GetLLM.py", line 5307, in main
        list_row_entries = [dpp[i],tune[0][i],tuneRMS[0][i],tune[1][i],... ]
    IndexError: list index out of range

A solution could be to check if the index exists before accessing.

The question is if it is senseless at all without the special BPM.

At another section GetLLM.GetFreeBetaFromAmp_Eq() returns the default value and the function GetLLM.filterBPM() raises because of that a SystemExit exception(sys.exit(1)). This exception will be catched by an 'except ALL' statement.
For now I added 'except SystemExit:sys.exit(1)' to let the program end.

            except SystemExit:
                sys.exit(1) 
            except:
                traceback.print_exc()
                pass

This solves the issue above since the part of the code will not be executed.
So I treat input files without an BPM next to AC dipole as invalid.

@Eothred @rogeliotomas @alangner
Is this correct or shall I better skip the sections which cause errors?
In this case there will be all other results which not depends on the special BPMs.

The getsuper test has been failing since 7. of December 2012. I tracked down this down using 'git bisect' to commit 0cb5a23 in Python_Classes4MAD, by @rogeliotomas.

The difference is fairly dramatic at first sight (not sure how serious). I have updated the ref files in commit 4dabef0 so that you can easily see the difference without having to run the tests.

@rogeliotomas @Eothred @alangner
Can you check the following?
In GetLLM in the section "Phase, Beta and coupling for non-zero DPP"(around 4930) the files getbetax_dpp_'+str(k+1)+'.out and getbetay_dpp_'+str(k+1)+'.out will be created.
I think the column names in the y file are wrong. Instead of BET>Y< and so on, there is BET>X<.

See the x-file:

filename = 'getbetax_dpp_'+str(k+1)+'.out'
...
tfs_file.add_column_names(["NAME","S","COUNT","BETX","ERRBETX","STDBETX","ALFX","ERRALFX","STDALFX","BETXMDL","ALFXMDL","

See the y-file(also BET>X<...):

filename = 'getbetay_dpp_'+str(k+1)+'.out'
...
tfs_file.add_column_names(["NAME","S","COUNT","BETX","ERRBETX","STDBETX","ALFX","ERRALFX","STDALFX","BETXMDL","ALFXMDL","

I assume all characters 'X' have to be replaced by 'Y'. Can you validate it?

@rogeliotomas @alangner @Eothred

What is the current version of iterative_correction?
1.43 is used by the GUI and 1.44 seems to be the latest in the x/Correction/Iterative/ directory.

Can someone provide a "correct" command which runs successfully?
I tried to run it with the GUI but it ends in errors. Version 1.44 corrects the path to job.iterative.correction.mask but later occurs a KeyError due to a BPM name which is not in the model.

Will this correction script still be used or shall I put the effort in the other correction scripts?

Opened to not forget about this issue. Here's the email exchange:

Hi Viktor,
this is one important "issue" to follow up in the c code (Drive).
We should probably just remove the BPMstatus function in drive
altogether. But I just wanted to send you this email so the issue
is not lost. You may open a ticket in the repository and address
it when you get to this code,
ciao
ro

-------- Original Message --------
Subject: Fwd: RE: drive question
Date: Mon, 10 Jun 2013 01:49:30 +0200
From: Thomas Bach [email protected]
To: Rogelio TOMAS GARCIA [email protected]

Hi Rogelio,

This is a reminder I found in my list of open mail topics.
I will forward them to you and "close" it on my side :)

Thomas

-------- Original Message --------
Subject: RE: drive question
Date: Fri, 7 Dec 2012 18:22:31 +0100
From: Rogelio Tomas Garcia [email protected]
To: Thomas Bach [email protected], Simon White [email protected]

Hi,
the BPM is ingnored if
amplitude[0] > 0 && label[i] == 1 && horizontalBpmCounter == i)

what means:
FFT amp < 0 (impossible)
or if peak-to-peak signal is smaller than 0.00001
or larger than 30000

or tune amp is smaller than background

or background is larger than 8.0

mmmh,
actually I doubt that these values make any sense now, because
before we did not have the powerful SVD clean and we trimmed these
values to remove bad BPMs... now we should re-check them
or maybe simply ignore these cuts inside Drive...
What do you think?

ciao
ro

From: Thomas Bach
Sent: 07 December 2012 16:22
To: Simon White; Rogelio Tomas Garcia
Subject: Re: drive question

If the question is still open: I do not know the exact behavior of the
fortran part. From the c part I would say that there are some more
reasons to not print BPMs.
Rogelio?

Thomas

On 07/12/2012 10:58, Simon White wrote:

Hi Thomas,

When you run drive with ISTUN=1, are the BPMs for which the tune is not
found within the specified window removed from the output?

Thanks,

Simon

@Eothred @rogeliotomas

The python script "MODEL/LHCB/fullresponse/generateFullResponse.parallel.py" uses an invalid name. First dot is not allowed.
The static analyses is not running because of that.

Sure, it runs but we should change it.
My suggestion is to replace the dot with an underscore:
"generateFullResponse_parallel.py"

As far as I found, the single place where it is used hardcoded is in ProgramVersions.properties:
"/afs/cern.ch/eng/sl/lintrack/Beta-Beat.src/CoreFiles/ProgramVersions.properties"

So it is no problem to change the entry there.

@Eothred If I shall change the name it would be better if you merge your branch with the master branch since you edited generateFullResponse. This would avoid merging conflicts.

@rogeliotomas @vimaier @tbachcern

argparse is an improved version over optparse for defining command line arguments. I noticed that argparse is at least available on the lxplus installations (running python 2.6.6). However, what I understand from Viktor is that it is not installed by default in python 2.6 (differing from 2.7). Will we have argparse available on all our servers?

getsex3000.out has no special signs ('*'/'$') in front of the column name line and column data type line:
...
NAME S AMP_20 ...
%s %le %le %le ...
...

There is a need to clean the natural tunes exactly as the tunes in the GUI(move the two markers in the Anaylsis panel and press clean).

I ran the WAnalyses in the BetaBeatGui and noticed that twiss_ac.dat will not be found. Here is the command with stderr output:

2013-05-23#14-44-18:
Name: WAnalysis
Command:
ssh cs-ccr-optics1 /usr/bin/python2.6 /afs/cern.ch/eng/sl/lintrack/Beta-Beat.src/GetLLM/getsuper.py --twissfile=/afs/cern.ch/work/v/vimaier/public/betabeatGui/temp/2013-05-23/models/LHCB1/model_with_ac/twiss.dat --output=/afs/cern.ch/work/v/vimaier/public/betabeatGui/temp/2013-05-23/LHCB1/Results/14-44-18_WANALYSIS_SUSSIX_5 --algorithm=SUSSIX --accel=LHCB1 --files=/afs/cern.ch/work/v/vimaier/public/betabeatGui/temp/2013-05-23/LHCB1/Measurements/Beam1@Turn@2012_10_12@13_21_57_097_0.new/Beam1@Turn@2012_10_12@13_21_57_097_0.sdds.new.new --madxbin=/afs/cern.ch/group/si/slap/bin/madx

2013-05-23#14-44-27:
Errors: Traceback (most recent call last):
  File "/afs/cern.ch/eng/sl/lintrack/Beta-Beat.src/GetLLM/getsuper.py", line 737, in 
    main(options,args)
  File "/afs/cern.ch/eng/sl/lintrack/Beta-Beat.src/GetLLM/getsuper.py", line 647, in main
    modeld=twiss(options.twiss+"/twiss_ac.dat")
  File "/afs/cern.ch/eng/sl/lintrack/Python_Classes4MAD/metaclass.py", line 90, in __init__
    f = open(filename, 'r')
IOError: [Errno 2] No such file or directory: './/twiss_ac.dat'

getsuper.py has an argument for twiss.dat-file(twissfile) and an argument for the path to twissfolder(twiss which default is './').
The command above sets the argument twissfile but not twiss. The result is that twiss_ac.dat will not be found.

To solve this problem I could remove the twiss argument and get the _ac version from twissfile by replacing .dat with _ac.dat.
twiss argument is implemented in GUI but not used. Here will the object be prepared:

final boolean successFul = getExternalPrograms().execute(
                    WAnalysis.prepareObject().setAccel(accel).setAlgorithm(analysisMethodAsString).setOutput(outputDirectory.toString()).setLogDirectory(outputDirectory).setTwissfile(twissfile)
                            .setFiles(files).setMadxbin(madxbin).create());

Shall I go ahead and fix it?

Adapt beta_from_phase so that it can calculate with 3BPMs again.

@asherman92 @rogeliotomas @alangner @Eothred
Alex noticed that the value of Window b2 is missing in the Drive.inp file. See picture below.

Further there are entries with default values. You cannot change CASE, KICK PERC. and LABEL RUN.

One line is missing to fix the issue with window b2:

final StringBuilder stringBuilderDriveInp = new StringBuilder();
stringBuilderDriveInp.append("KICK=").append(analysisSettingsData.getStartTurn()).append(linebreak);
stringBuilderDriveInp.append("CASE(1[H], 0[V])=1").append(linebreak);
stringBuilderDriveInp.append("KPER(KICK PERCE.)=0.5").append(linebreak);
stringBuilderDriveInp.append("TUNE X=").append(analysisSettingsData.getTuneX()).append(linebreak);
stringBuilderDriveInp.append("TUNE Y=").append(analysisSettingsData.getTuneY()).append(linebreak);
stringBuilderDriveInp.append("PICKUP START=").append(analysisSettingsData.getStartBpm()).append(linebreak);
stringBuilderDriveInp.append("PICKUP END=").append(analysisSettingsData.getEndBpm()).append(linebreak);
stringBuilderDriveInp.append("ISTUN=").append(analysisSettingsData.getTuneWindow()).append(linebreak);
stringBuilderDriveInp.append("LABEL RUN (1[yes])=0").append(linebreak);
stringBuilderDriveInp.append("WINDOWa1=").append(analysisSettingsData.getWindowA1()).append(linebreak);
stringBuilderDriveInp.append("WINDOWa2=").append(analysisSettingsData.getWindowA2()).append(linebreak);
stringBuilderDriveInp.append("WINDOWb1=").append(analysisSettingsData.getWindowB1()).append(linebreak);
// ### Missing line
stringBuilderDriveInp.append("WINDOWb2=").append(analysisSettingsData.getWindowB2()).append(linebreak);
// ### Missing line
stringBuilderDriveInp.append("NATURAL X=").append(analysisSettingsData.getNaturalTuneX()).append(linebreak);
stringBuilderDriveInp.append("NATURAL Y=").append(analysisSettingsData.getNaturalTuneY()).append(linebreak);

Shall I add the line to write Window b2 to Drive.inp?

What is with the default values? Shall I add them to the settings screen?

Drive outputs also spectrum(all *.x and *.y files) which is not used currently by GUI.
It is proposed to add a new tab(Frequency) to the Analysis panel which displays the spectrum. The old chart remains in "TimeSpace" tab:

coupling f1001 in GetLLM.GetCoupling(1|2) has to be checked with simulations.

@rogeliotomas @Eothred @alangner

I noticed that SegmentBySegment has also a second dot in the filename.
The only occurence is in the ProgramVersions.properties file.
Can I rename it to 'SegmentBySegment_0_33.py'?
Further the folder SegmentBySegment contains a lot of versions of SegmentBySegment. See:
https://github.com/pylhc/Beta-Beat.src/tree/master/SegmentBySegment

I would like to delete all old versions and keep the newest version as 'SegmentBySegment.py'.

Below you can see all occurences of 'SegmentBySegment_*'.
I guess the only relevant file is ProgramVersions.properties(I replaced dot with underscore already).

The folder CoreFiles contains some setup.txt files.
Are these files for the betabeatGUI and outdated? Can we delete them?

Beta-Beat.src
    CoreFiles
        #setup.txt#
            114: VALUE=/SegmentBySegment/SegmentBySegment_0.29.py 
        ProgramVersions.properties (2 matches)
            74: sbsmeas = /SegmentBySegment/SegmentBySegment_0_33.py 
            75: SegmentBySegment = SegmentBySegment/SegmentBySegment_0_33.py 
        setup_current.txt
            109: VALUE=/SegmentBySegment/SegmentBySegment_0.1.py 
        setup.txt
            121: VALUE=/SegmentBySegment/SegmentBySegment_0.29.py 
    SegmentBySegment
        README
            1: python SegmentBySegment_0.0.py ...
    WWW
        jnlpRun
            LHC
                2011
                    IONS_1m1m1m3m_28-10-11
                        README (4 matches)
                            8:  ... BySegment/SegmentBySegment_0.32_dev.py... 
                            11:  ... BySegment/SegmentBySegment_0.32_dev.py... 
                            15:  ... BySegment/SegmentBySegment_0.32_dev.py ... 
                            18:  ... BySegment/SegmentBySegment_0.32_dev.py ...

@rogeliotomas @alangner

Rogelio ran the new SegmentBySegment.py(sbs) script and received following error:

Traceback (most recent call last):
  File "/afs/cern.ch/work/v/vimaier/public/Beta-Beat.src/SegmentBySegment/SegmentBySegment.py", line 2349, in 
    return_value = main(options)
  File "/afs/cern.ch/work/v/vimaier/public/Beta-Beat.src/SegmentBySegment/SegmentBySegment.py", line 500, in main
    getAndWriteData(namename,phases,betah,betav,disph,dispv,couple,chromatic,twisstwiss,modelcor,normal_pro,back_pro,savepath,elementswitch,accel)
  File "/afs/cern.ch/work/v/vimaier/public/Beta-Beat.src/SegmentBySegment/SegmentBySegment.py", line 1550, in getAndWriteData
    delta_phase = (phasey.PHASEY[phasex.indx[name]] - phasex.PHASEY[phasey.indx[first_bpm]]) %1      
AttributeError: twiss instance has no attribute 'PHASEY'

The old sbs ran without errors.
Error is caused by this new line:

delta_phase = (phasey.PHASEY[phasex.indx[name]] - phasex.PHASEY[phasey.indx[first_bpm]]) %1 

There are two logical errors.

• phasey.PHASEY[phasex.indx[name]] ; phasey-file but phasex index?
• phasex.PHASEY[phasey.indx[first_bpm]]) ; should probably be PHASEX

The second error caused the exception since phasex file has no PHASEY column.

I do not know which delta_phase is supposed to be calculated here. For now I changed the calculation to delta_phase_x between current bpm(name) and first_bpm:

delta_phase = abs(phasex.PHASEX[phasex.indx[name]] - phasex.PHASEX[phasex.indx[first_bpm]]) %1 

Note that I took the absolute value. You have to pay attention with modulo.
Consider following code and output

print 0.00828579 - 0.0104637533333      -->       -0.0021779633333
print (0.00828579 - 0.0104637533333) %1   -->       0.997822036667 !complementary value to delta and 1
print abs(0.00828579 - 0.0104637533333) %1 -->   0.0021779633333 !correct delta

These are real values from PHASEX and PHASEY.

Another thing. While debugging this case I noticed that phasex was the file getphasex_free.out and phasy was getphasey.out (no free version).
Another bug was here(checking of getphase_free.out):

    try:
        filephasex=twiss(path+"/getphasex_free.out")
    except:
        filephasex=twiss(path+"/getphasex.out")    

    try:
        filephasey=twiss(path+"/getphasey.out")
    except:
        filephasey=twiss(path+"/getphasey_free.out")
    try:
        filephasextot=twiss(path+"/getphasetotx_free.out")
    except:
        filephasextot=twiss(path+"/getphasetotx.out")

    try:
        filephaseytot=twiss(path+"/getphasetoty_free.out")
    except:
        filephaseytot=twiss(path+"/getphasetoty.out")

Fixed this bug and replaced the excepts with except (IOError, ValueError).

@rogeliotomas @Eothred @alangner

I want to add 'filename' to twiss so I can see to which file a twiss instance belongs during debugging.
Further GetLLM has a list with twiss instances and a list with corresponding filenames. The latter would be unnecessary.

I would need to add a single line into the constructor:

def __init__(self, filename, dictionary=None):
...
        self.filename = filename

Actually I changed metaclass already in my local copy. Since I want to use it in GetLLM I need the change global.

I'm not familiar with developing on Python_Classes4MAD. I don't want to make changes on AFS. I prefer the git way:
I would push on a not checked out "dev" branch and merge the changes into the master branch.

May I go on?

By the way I would print a message if a line is not parsed. This would be helpful. See issue #19.

I replaced almost all pass statements after excepts with traceback.print_exc().
Thus you can see all raised exceptions. get_ip_2 raises some interesting exceptions when I run GetLLM with "run_with_ac":
/afs/cern.ch/eng/sl/lintrack/Beta-Beat.src/GetLLM/test/data/run_with_ac

Traceback (most recent call last):
  File "C:\eclipse_4_2_2_python\workspace\Beta-Beat.src\GetLLM\GetLLM.py", line 1306, in calculate_beta_from_amplitude
    beta_d.y_ratio_f = beta_d.y_ratio_f / (len(arcbpms) - len(skipped_bpmyf))
ZeroDivisionError: integer division or modulo by zero

Traceback (most recent call last):
  File "C:\eclipse_4_2_2_python\workspace\Beta-Beat.src\GetLLM\algorithms\helper.py", line 2227, in get_ip_2
    rt2j = math.sqrt(a_l*a_r*sin(dpsi)/(2*L))
ValueError: math domain error

C:\Python26\lib\site-packages\numpy\core\_methods.py:57: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret / float(rcount)

Traceback (most recent call last):
  File "C:\eclipse_4_2_2_python\workspace\Beta-Beat.src\GetLLM\algorithms\helper.py", line 2227, in get_ip_2
    rt2j = math.sqrt(a_l*a_r*sin(dpsi)/(2*L))
ValueError: math domain error

Traceback (most recent call last):
  File "C:\eclipse_4_2_2_python\workspace\Beta-Beat.src\GetLLM\algorithms\helper.py", line 2227, in get_ip_2
    rt2j = math.sqrt(a_l*a_r*sin(dpsi)/(2*L))
ValueError: math domain error

Traceback (most recent call last):
  File "C:\eclipse_4_2_2_python\workspace\Beta-Beat.src\GetLLM\algorithms\helper.py", line 2227, in get_ip_2
    rt2j = math.sqrt(a_l*a_r*sin(dpsi)/(2*L))
ValueError: math domain error

The reason for the "ValueError: math domain error" errors is that sin(dpsi) is negative and math.sqrt tries to extract the root from a negative number.
Here will dpsi be calculated:

if plane == 'H':
...
    if bpm_names.index(bpmr) > bpm_names.index(bpml):
        dpsi = 2*np.pi*beam_direction*(t_f.MUX[t_f.indx[bpmr]]-t_f.MUX[t_f.indx[bpml]])
    else:
        dpsi = 2*np.pi*(Q+beam_direction*(t_f.MUX[t_f.indx[bpmr]]-t_f.MUX[t_f.indx[bpml]]))
elif plane == 'V':
...
    if bpm_names.index(bpmr) > bpm_names.index(bpml):
        dpsi = 2*np.pi*beam_direction*(t_f.MUY[t_f.indx[bpmr]]-t_f.MUY[t_f.indx[bpml]])
    else:
        dpsi = 2*np.pi*(Q+beam_direction*(t_f.MUY[t_f.indx[bpmr]]-t_f.MUY[t_f.indx[bpml]]))

The reason could be bad input files but the exceptions could also be an indicator for a failure.
Has to be checked...

@rogeliotomas @Eothred @alangner

We have four different versions of Drive(FASTER, Harry, Massimo, Mcintosh).
Do we need all of them?
Can we "clean"(delete) them?

GUI only runs Harry version.

Further the directory names include a dot. It is not possible to treat them as Python packages. I want to implement a test for Drive with the Python testing framework.

Can I rename DRIVE_src.Harry to drive or create the folder drive and copy the content from Harry into the new folder?

For GUI I only would have to change ProgramVersions.properties.

From errors in the logfile I saw that GetLLM often raises following error:

Traceback (most recent call last):
  File "/afs/cern.ch/eng/sl/lintrack/Beta-Beat.src/GetLLM/GetLLM.py", line 5325, in 
    higher_order=options.higher)
  File "/afs/cern.ch/eng/sl/lintrack/Beta-Beat.src/GetLLM/GetLLM.py", line 5280, in main
    [invarianceJx,invarianceJy,tune,tuneRMS,dpp]=getkick(files,MADTwiss)
  File "/afs/cern.ch/eng/sl/lintrack/Beta-Beat.src/GetLLM/GetLLM.py", line 2289, in getkick
    tunex.append(x.Q1)
AttributeError: twiss instance has no attribute 'Q1'

I debbuged it and noticed that the twiss instance had no Q1 attribute although the file(*_linx) had this attribute.
Here is the line:
@ Q1 FLOAT 3.096808e-01

Instead of "%le" is there "FLOAT".
That is the reason why twiss() does not parse the line and the resulting instance has no attribute Q1.

Here you can take a look at the file:

/afs/cern.ch/work/v/vimaier/public/betabeatGui/temp/2013-05-24/LHCB1/Measurements/Beam1@Turn@2012_03_15@09_28_21_723_0.new/Beam1@Turn@2012_03_15@09_28_21_723_0.sdds.new.new_linx

I do not know the reason for the replacement. It occurs not every time.

I will track it down.

The file "x/MODEL/LHCB/fullresponse/LHCB1AllLists_couple.json" contains the entry:

"coupling_knobs": [
    "b1_re_ip1_knob",  
    "b1_im_ip1_knob"
  ]

The GUI calls the function findLogicalNamesByMadStrengthNames() and tries to convert the MadStrengthNames "b1_re_ip1_knob" and "b1_im_ip1_knob" into corresponding logical names.
The problem is that the function cannot find logical names for these MadStrengthNames and the Task createKnob crashes. I added a check and a print in case of failure:

14:19:21 - LsaServices => The number of translated correctors is not equal to input. Contact expert. Knob will not be created!
 CorrectorNames      MappedNames
 b1_re_ip1_knob             null
 b1_im_ip1_knob             null

@Eothred @alangner @rogeliotomas
Any idea why the LSA function does not find logical names?

correct_coupleDy.py and coupling_knobs will be always used in the automatic correction of the GUI.

The script MODEL.LHCB.fullresponse.generateFullResponse_parallel.py produces the file iter.madx three times for chromatic coupling, chromatic and then beta. The file will every time be overwritten.
The last iter.madx for beta can probably be deleted in the end.

Test which compares output of SegmentBySegment.py script with SBS version 3.3.

@Eothred @rogeliotomas @alangner

I encountered now severeal times:

try:
    from metaclass25 import *
except:
    from metaclass import *

I looked at metaclass25 and it seems to be an older version than the one without suffix '25'.
Can someone explain why does this module exist? Is it maybe only a version number and I can remove it?

Edit: misspelling

the team considers to change the default value of "SVD clean sing val cut" from 57 to 10.

In both files the first columns are equal(bpm name). This is probably wrong. See below:

Hi,

Me again. I fixed the output in both files:

From(getf2000x.out)
@ FILES %s ""* NAME S F2000L F2000EL P2000L P2000EL F2000N F2000EN P2000N P2000EN
$...

and8getf2000y.out)
@ FILES %s "
"* NAME S F2000L F2000EL P2000L P2000EL F2000N F2000EN P2000N P2000EN
$...

to
@ FILES %s ""
* NAME S F2000L F2000EL P2000L P2000EL F2000N F2000EN P2000N P2000EN
$ ...

(No line where file names are added. Shall I add src file names to getf2000*.out?)

But there is another weird thing:

@ GetLLMVersion %s "V2.38b PRO"
@ MAD_FILE %s "./data\getsuper_data2\input/model/twiss.dat"
@ FILES %s ""

•        NAME                 S                        F2000L                     .    .    .  
  

  $ %s %le %le
  BPMWB.4R5.B1 BPMWB.4R5.B1 0.00900454358056
  BPM.12L3.B1 BPM.12L3.B1 0.0118503051275
  BPM.25R7.B1 BPM.25R7.B1 0.00844995307049
  BPM.16L2.B1 BPM.16L2.B1 0.00458893143187
  .
  .
  .

The second column should contain float numbers instead of strings. Actually they are unnecessary since the names are the same as in the first column.
Here is the code:

result_f2000=Getquadrupole(MADTwiss,names,plane,phaseylist[0],betayalist[0])
for name in names:

    loc=result_f2000[name][8]

    f2000y.write(name+" "+str(loc)+" "+str(result_f2000[name][0])+.    .    .

result_f2000 is a dictionary with(key-->value) bpm-name --> list-of-values-with-9-entries.
A typical value list(key == BPM.10L3.B1):
list: [1.6726061495524001e-07, -0.708785425037645, 0, 0, 0, 0, 0, 0, 'BPM.10L3.B1']

loc is the entry with index 8(last one) and this is the bpm-name again.

I assume loc should be something else.

I will also track it as ticket.

Cheers,
Viktor

getsuper does not work when we want to analyse kick data instead of ac dipole data. Somehow GetLLM is still producing _free files (??), and getsuper then thinks that we are analysing AC dipole data but cannot find twiss_ac.dat in the model. The logic here seems completely broken.

@vimaier I have a test added in my own fork, should I add it to the github master? It is currently failing until this bug is fixed..

I started to clean the SegmentBySegment dir.
There were a lot of files containing only different versions. Therefor we have git now.
Before I removed a file I checked that the filename did not appear anywere.
Here are the remaining files(some of them are renamed to avoid version number):
│ getfterms.py
│ getfterms_RHIC.py
│ gplot.IP2IP8.mask
->│ gplot.IP8B2.mask (not used anywhere)
│ gplot.mask
│ gplot_RHIC.mask
->│ job.InterpolateBetas.0_1_RHIC.mask (not used anywhere)
│ job.InterpolateBetas.mask
│ SegmentBySegment.py
->│ watch.py (not used anywhere)
│ init.py

@rogeliotomas @Eothred @alangner
Does somebody have information about the files which are not used?

GUI has currently following three correction methods: BETA, COUPLING, CHROMATIC_COUPLING

If a file/an entry will be added to the CorrectionsPanel(last tab where one can open the KnobPanel) then the GUI loads either changeparameters.tfs for BETA or changeparameters_couple.tfs for COUPLING and CHROMATIC_COUPLING but never changeparameters_>>chrom<<couple.tfs (without < and >).

The script correct_ChromCoup.py produces changeparameters_chromcouple.tfs.

@rogeliotomas @Eothred
Shall I adapt the GUI to load changeparameters_chromcouple.py?

@rogeliotomas @Eothred @alangner
I found a small(?) discrepancy in GetLLM(around 4739-4760)
The for-loop iterates over "ListOf>Non<ZeroDPPY" but len(ListOfZeroDPPY) will be written into the third column:

for j in ListOfNonZeroDPPY:    
...        
            for i in range(0,len(bpms)):
...         
                list_row_entries = ['"'+bn1+'"',bns1, len(ListOfZeroDPPY), codpp[bn1][0], codpp[bn1][1], MADTwiss.Y[MADTwiss.indx[bn1]], MADTwiss.MUY[MADTwiss.indx[bn1]] ]

It looks like it should be len(ListOfNonZeroDPPY).

Can you checkt it?

@Eothred @alangner @rogeliotomas

The following file contains two lines which will not be parsed by metaclass.twiss:

/afs/cern.ch/work/v/vimaier/public/corrupt_model/Model2/twiss_elements.dat

sys.stderr from metaclass.twiss(twiss_elem):

Did not parse line ( "LHCB1$END" 23519.36962 11.00000.... ) in  twiss_elements.dat
Did not parse line ( "LHCB1$START" 23519.36962 11.0000.... ) in  twiss_elements.dat

This are lines 3144 and 3145.

However "LHCB1$END" and "LHCB1$START" do not sound like BPM names. It seems like some variables were not replaced properly in MAD-X.

@Eothred , you are developing on MAD-X, right? Do you know, who could check it?

Thomas suggested to include additional information(date and arguments) to the output files to track the origin of the files.

The following condition checks if there are linx files with dpp!=0 and no linx files with dpp==0.
If it is True then linx and liny files with dpp!=0 will be treated as if they would have dpp==0.

if (len(twiss_holder.non_zero_dpp_x) != 0) and (len(twiss_holder.zero_dpp_x) == 0):
        twiss_holder.zero_dpp_x = twiss_holder.non_zero_dpp_x
        twiss_holder.zero_dpp_y = twiss_holder.non_zero_dpp_y
        with_linx_for_zero_dppx = True
        with_liny_for_zero_dppy = True
        with_liny_for_nonzero_dppy = False
        with_linx_for_nonzero_dppx = False
        print "Previous warning suppressed, running in chromatic mode"

twiss_holder.non_zero_dpp_x(or y) will not be accessed because the corresponding flag is False except of the calculation of getkick.out.
All files will be taken here to calculate kick:

files = [twiss_holder.zero_dpp_x + twiss_holder.non_zero_dpp_x, twiss_holder.zero_dpp_y + twiss_holder.non_zero_dpp_y]
...
getkick(files, MADTwiss)

All twiss files will be used to calculate kick. If the condition above is true then the input files will be twice in the list.
getkick.out would have rows with same content since one row will be calculated from one linx and liny file of one src file.

Our run directories cover this case.

The assignment of empty lists fixed it:

if (len(twiss_holder.non_zero_dpp_x) != 0) and (len(twiss_holder.zero_dpp_x) == 0):
        twiss_holder.zero_dpp_x = twiss_holder.non_zero_dpp_x
        twiss_holder.zero_dpp_y = twiss_holder.non_zero_dpp_y
>>>
        twiss_holder.non_zero_dpp_x = []
        twiss_holder.non_zero_dpp_y = []
>>>
        with_linx_for_zero_dppx = True
        with_liny_for_zero_dppy = True
        with_liny_for_nonzero_dppy = False
        with_linx_for_nonzero_dppx = False
        print "Previous warning suppressed, running in chromatic mode"

The following statements are completely useless except of the assignment of tune_d.q1 and tune_d.q2.
Nothing else will be used further:

if twiss_d.has_non_zero_dpp_y() and twiss_d.has_non_zero_dpp_x():
    if len(twiss_d.non_zero_dpp_x) != len(twiss_d.non_zero_dpp_y):
        raise ValueError("list of dppx is not equal list of dppy")
    for j in range(len(twiss_d.non_zero_dpp_x)):
        dpop = float(twiss_d.non_zero_dpp_x[j].DPP)
        list_with_single_twiss_x = []
        list_with_single_twiss_y = []
        list_with_single_twiss_x.append(twiss_d.non_zero_dpp_x[j])
        list_with_single_twiss_y.append(twiss_d.non_zero_dpp_y[j])
        ### coupling
        try:
            mad_twiss.Cmatrix()
        except:
            pass
        if getllm_d.accel == "SPS" or "RHIC" in getllm_d.accel:
            #TODO: check parameter. Q seems missing in calls get_phases (vimaier)
            plane = 'H'
            [phasexp, tune_d.q1, MUX, bpmsx] = algorithms.helper.get_phases(getllm_d, mad_twiss, pseudo_list_x, plane)
            plane = 'V'
            [phaseyp, tune_d.q2, MUY, bpmsy] = algorithms.helper.get_phases(getllm_d, mad_twiss, pseudo_list_y, plane)
            [fwqw, bpms] = algorithms.helper.GetCoupling2(mad_twiss, pseudo_list_x, pseudo_list_y, tune_d.q1, tune_d.q2, phasexp, phaseyp, getllm_d.beam_direction, getllm_d.accel)
        elif getllm_d.num_beams_for_coupling == 1:
            [fwqw, bpms] = algorithms.helper.GetCoupling1(mad_twiss, list_with_single_twiss_x, list_with_single_twiss_y, tune_d.q1, tune_d.q2)
        elif getllm_d.num_beams_for_coupling == 2:
            [fwqw, bpms] = algorithms.helper.GetCoupling2(mad_twiss, list_with_single_twiss_x, list_with_single_twiss_y, tune_d.q1, tune_d.q2, phasexlist[j], phaseylist[j], getllm_d.beam_direction, getllm_d.accel)
            if getllm_d.with_ac_calc:
                [fwqw, bpms] = algorithms.helper.getFreeCoupling(tune_d.q1f, tune_d.q2f, tune_d.q1, tune_d.q2, fwqw, mad_twiss, bpms)
        else:
            raise ValueError('Number of monitors for coupling analysis (option -n) should be 1 or 2.')
        fwqw['DPP'] = dpop

The code could be minimized to:

if twiss_d.has_non_zero_dpp_y() and twiss_d.has_non_zero_dpp_x():
    if len(twiss_d.non_zero_dpp_x) != len(twiss_d.non_zero_dpp_y):
        raise ValueError("list of dppx is not equal list of dppy")
    for j in range(len(twiss_d.non_zero_dpp_x)):
        ### coupling
        try:
            mad_twiss.Cmatrix()
        except:
            pass
        if getllm_d.accel == "SPS" or "RHIC" in getllm_d.accel:
            #TODO: check parameter. Q seems missing in calls get_phases (vimaier)
            plane = 'H'
            [phasexp, tune_d.q1, MUX, bpmsx] = algorithms.helper.get_phases(getllm_d, mad_twiss, pseudo_list_x, plane)
            plane = 'V'
            [phaseyp, tune_d.q2, MUY, bpmsy] = algorithms.helper.get_phases(getllm_d, mad_twiss, pseudo_list_y, plane)

Q1 and Q2 will only be assigned if accel is SPS or RHIC. The invocations of get_phases with these accelerators are wrong(Tune is missing, see issue #23 ).
Not sure but it seems the last time the scripts were running with these accelerators is long time ago.

@rogeliotomas @Eothred @alangner
Does it have any effect to call mad_twiss.Cmatrix() multiple times like in the snippet above?

Seems that Drive.inp will never be found.

Answer from Ryoichi:

First, GetCoupling1 and 2 are written by Masamitsu so he might remember
about these better than me.

As you know, this part judges whether Q is larger than 0.5 or not
and, if so, converts Q from Sussix (which is always below 0.5) to
1-Q. Just to do this, we can read Q or mu from MAD-Twiss and make
a judgement based on the model. This is not perfect if the machine
and model are very far but it's hard to believe this happens.

Now, wether we need this conversion or not (I mean what happens
if Qx,Qy0.7 but doing the analysis with Qx,Qy0.3, for example
assuming RHIC), I'm not 100% sure. If you want, I can go back to
equations and check. Or, somebody may be able to do a simple test.
If you just want not to read Drive.inp, there should be a way
to do this conversion with model like above or something else.

One more comment. fracx and fracy are used to define sign_QxmQy
and this sign_QxmQy is used to calculate the global coupling
in the end. I didn't do much about the global coupling and
don't remember well. But all we need must be sin(pi(Qx-Qy)) and
sin(pi(Qx+Qy)) so I wonder why we need sing_QxmQy. In other ways,
there may be a way to do the calculation of the global coupling
just from abs(Qx-Qy).

getIPx
getIPy.out
getIPfromphase
getIPx_free
getIPy_free
getIPx_free2
getIPy_free2
getIPfromphase_free
getIPfromphase_free2

In the files above is a line missing which describes the data type of the columns:
$ %s %le %le ...

Example file:
@ GetLLMVersion %s "V2.38b PRO"
@ MAD_FILE %s "./data\getsuper_data3\input/model/twiss.dat"
@ FILES %s "C:\eclipse_4_2_2_python\workspace\Beta-Beat.src\GetLLM\test\data\getsuper_data3\input\src_files\bpms.gz"

• NAME BETY BETYSTD BETYMDL ALFY ALFYSTD ALFYMDL BETY* BETY_STD BETY_MDL SY* SY_STD SY_MDL rt(2JY) rt(2JY)STD
  "IP1" 1.06888574865 0.0 1.03156713291 0.10598680812 0.0 -0.0282988589718 1.05701211645 0.0 1.03074168877 0.112029340367 0.0 -0.0291688136868 0.0147407384622 0.0
  "IP5" 1.0733434976 0.0 0.960333653086 -0.290894896094 0.0 -0.00999424879369 0.989603414435 0.0 0.960237739731 -0.287870582416 0.0 -0.00959685487197 0.0151956966971 0.0
  "IP8" 2.96997751783 0.0 2.99222794193 -0.214346654973 0.0 -0.0416072272689 2.8395173458 0.0 2.9870568645 -0.608641044809 0.0 -0.124283153826 0.0149147817432 0.0
  "IP2" 227.171961036 0.0 10.3926183638 0.310824915702 0.0 0.0164079919896 207.157987031 0.0 10.3898211933 64.389863856 0.0 0.170476102913 0.0074457587808 0.0

  END Example file

Due to timeout the test fails sometimes on testserver.
It runs the modified and the valid drive and compares the output. I will store only the output of the valid file and compare the outcome of the modified file with ndiff. The run time should be halved then.

If the values TUNE X and TUNE Y in Drive are bigger than 0.5 or smaller than -0.5 then they have to be adapted:
if value > 0.5:
--value
print "Automatic correction of tune from %f to %f." % value+1, value
else if value < -0.5:
++value
print "Automatic correction of tune from %f to %f." % value-1, value

Since Simons feature needs the natural tune it is good to have it in the getkick*.out files.
The values can be obtained from linx and liny(Drive output).