bootlin / elixir Goto Github PK

Hi, I've been working on setting up my own copy of Elixir with the Linux kernel and discovered that the README.md file is out of date and no longer reflects how Elixir is set up and configured.

1. Elixir also depends on the jinja2 and pygments python libraries which are not mentioned in the README.

2. The example Apache configuration uses the LXR_DATA_DIR and LXR_REPO_DIR for configuration (just like update.py), however web.py overrides those environment variables based on what's in the undocumented LXR_PROJ_DIR environment variable. This also has the side effect that web.py requires the data and repo directories to be in a specific (and undocumented) structure.

   So consequently, the example directory structure of LXR_DATA_DIR=/srv/elixir-data, LXR_REPO_DIR=/srv/git/linux will not work (as it differs from web.py's assumed directory structure).

3. The URL scheme in the example Apache configuration also differs from what web.py uses. For example, to access v4.10 of the linux kernel source code, the example configuration uses ^/source, ^/ident, etc. whereas web.py uses ^/linux/v4.10/source, ^/linux/v4.10/ident, etc.

I was searching for kernfs_drain() but this symbol is not known.

http://elixir.free-electrons.com/linux/latest/source/fs/kernfs/dir.c#L456

I think the issue is the attributes after the declaration, __releases(&kernfs_mutex) __acquires(&kernfs_mutex). There are other functions with such attribute missing, e.g. debugfs_use_file_finish() it knows of the EXPORT, but not the function. fuse_send_writepage() is referenced, but not defined.

Attempting to update a repo that contains files with spaces in their names results in the following output`

stdk@stdk:~/dev/elixir$ ./update.py
Found 2 new tags
initial: Traceback (most recent call last):
  File "./update.py", line 138, in <module>
    newBlobs = updateBlobIDs (tag)
  File "./update.py", line 46, in updateBlobIDs
    hash, filename = blob.split (b' ')
ValueError: too many values to unpack (expected 2)

Repository that produces such behaviour can be created in the following way:

# git init
# echo "int main() { return 0; }" > "Spaces included.c"
# git add "Spaces included.c"
# git commit -m "Initial"

Hello,

I got a temporary "Attribute error" on http://elixir.free-electrons.com, probably when the database was updated to index the v4.10 tag.

Unfortunately, I can't copy the text any more. I just have the attached screenshot.

Michael.

Anything after this line is not parsed and hence not in the cross-reference:

http://elixir.free-electrons.com/linux/v3.11.10/source/fs/namespace.c#L974

E.g. a search for the identifier vfs_kern_mount shows references to line 775 and 805 in namespace.c, but actually it's called on line 1998 and 2691 and 2761 and you get the idea.

I built a cross referencer site for kernel and glibc using elixir.
But I noticed that the load time of a page is very long about 500ms.
So can we support pre generate output of a source file to a static file, then we can load it without dynamic generate , that will speed up the visit time

Hi!

I was looking for all references of a certain struct in the Linux Kernel when I found out that
exlixir does not list all of them.
If I search for super_block, for example, I do get a list of references:
https://elixir.bootlin.com/linux/v4.19-rc6/ident/super_block
It works as expected for typedefs as well:
https://elixir.bootlin.com/linux/v4.19-rc6/ident/journal_t

However, if I search for inode, I just get the definition of struct inode:
https://elixir.bootlin.com/linux/v4.19-rc6/ident/inode

I suppose this is a bug in the update mechanism.

Regards,
Alex

The declaration of the "jiffies" symbol is not listed in
https://elixir.bootlin.com/linux/latest/ident/jiffies

...though this symbol is declared in https://elixir.bootlin.com/linux/latest/source/include/linux/jiffies.h:
extern unsigned long volatile __cacheline_aligned_in_smp __jiffy_arch_data jiffies;

Such complex declarations should be supported too ;)

Right now, the only way to know which version is currently selected is through the URL. The currently selected version should be displayed in the page itself.

Right now, you can't copy-paste code because the line numbers are included in the selection. This could be fixed by displaying the line numbers and the code inside a two-column invisible table but proper alignment must be ensured.

Elixir is successfully used by the DPDK project:
http://code.dpdk.org

Thank you Bootlin and contributors!

Sometimes for a function I get sent to the line with the initial { rather than the function header. At the moment, that occurs with of_iomap. The entry in drivers/of/address.c links to line 761, but the function starts on line 760

http://elixir.free-electrons.com/linux/latest/source/drivers/of/address.c#L761

There are some other function links for of_iomap, but they go to the right place.

The feature I liked the most of the old system was the ability to diff through the versions particularly when trying to find when something was changed. While I like the layout of the new system since you can fit a bunch more versions, it's not usable to me without the diff feature.

Thanks.

How do you think about to support the display and advanced navigation for call graphs besides of repeated searches for a single function identifier?

It gets error while acceess the eos project with follow url
http://192.168.50.159/lxr-eos/latest/source

But access the zephyr project is good.
http://192.168.50.159/zephyr/latest/source

The web page report:

Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator at root@localhost to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log.

I got the follow error from apache error_log:

[Fri Jun 29 20:47:49.231913 2018] [http:error] [pid 775] [client 192.168.50.97:51215] AH02429: Response header name '<!--' contains invalid characters, aborting request

My environment:
OS: centos7
web server: apache 2.4.6
my apahce config file:
[root@localhost httpd]# cat sites-available/lxr.eos.conf

<VirtualHost *:80>
    ServerName lxr.eos.com
    DocumentRoot "/data/app/workspace/git/elixir/http"

    RewriteEngine on
    RewriteRule "^/$" "/lxr-eos/latest/source" [R]
    RewriteRule "^/.*/(source|ident|search)" "/web.py" [PT]
    <Directory "/data/app/workspace/git/elixir/http">
        Options Indexes FollowSymLinks ExecCGI
        AllowOverride All
        AddHandler cgi-script .cgi .py
        Require all granted
        SetEnv PYTHONIOENCODING utf-8
        SetEnv LXR_PROJ_DIR /data/home/jiaxi/workspace/job
    </Directory>
</VirtualHost>

project directory(contains zephyr and eos):
[root@localhost httpd]# ls ~jiaxi/workspace/job/zephyr/
data repo

[jiaxi@localhost elixir]$ ls ~/workspace/job/lxreos/
data repo

any suggestion is good, thanks.

Hello,

This URL used to work:
http://lxr.free-electrons.com/source/arch/sh

But not this one:
http://elixir.free-electrons.com/source/arch/sh

The trailing / is required in Elixir. Would it be possible to support the absence of "/" in Elixir too?

Don't do this if this is expensive to implement or hurts the simplicity of the code. I'm not even sure directory URLs without the trailing / were used by LXR users (as the links provided by LXR did contain the / character at the end.

Michael.

My web browser (chrome 57.0.2987.98) fail to display lxr.free-electrons.com correctly.
My system is a Debian 9 and I get the version from apt:
deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main

Everything works fine with firefox on the same machine

.

LXR has a convenient http://lxr.free-electrons.com/ident that is friendly for bookmarking. Every new stable release updates this URL. However elixir doesn't have that and I did not find anyway of having a bookmark to identifier search that won't require me to edit bookmarks on every release.

Firefox 61.0 64-bit on Debian testing

If I load https://elixir.bootlin.com/linux/latest/source (or any other page of elixir),
move the source tree or try to open a folder using my finger on the touchscreen, the whole page turns white and there's nothing to do but reload the page and use a mouse.

Debian Web Server

Need to migrate to debian testing to get python3.5, Doesn't work with python3.4.

As root :
root@debian:~ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
root@debian:~ git clone https://github.com/free-electrons/elixir.git
root@debian:~ aptitude install python3 python3-pygments python3-jinja2 exuberant-ctags kamailio-berkeley-modules python3-bsddb3

root@debian:~ mkdir /root/data
root@debian:~ export LXR_REPO_DIR=/root/linux
root@debian:~ export LXR_DATA_DIR=/root/data

root@debian:~ cd elixir
root@debian:~ ./update.py #changed update.py to create only the last tag
root@debian:~ mv /root/linux/ /srv/elixir-data/linux/repo
root@debian:~ mv /root/data /srv/elixir-data/linux/
root@debian:~ mv /root/elixir/ /usr/local/

Add to apache2.conf :

<Directory /usr/local/elixir/http/>
    Options +ExecCGI
    AllowOverride None
    Require all granted
    SetEnv PYTHONIOENCODING utf-8
    SetEnv LXR_PROJ_DIR /srv/elixir-data
</Directory>

AddHandler cgi-script .py

<VirtualHost *:80>
    ServerName MY_LOCAL_IP
    DocumentRoot /usr/local/elixir/http

    RewriteEngine on
    RewriteRule "^/$" "/linux/latest/source" [R]
    RewriteRule "^/.*/(source|ident|search)" "/web.py" [PT]
</VirtualHost>

root@debian:~ a2enmod rewrite
root@debian:~ a2enmod cgi
root@debian:~ systemctl restart apache2

Searching for "SND_SOC_DAPM_POST_PMU" yields a number of hits:

http://elixir.free-electrons.com/linux/v4.10/ident/SND_SOC_DAPM_POST_PMU

Clicking on "line 217" from "include/sound/soc-dapm.h" appears to take me to line 217:

http://elixir.free-electrons.com/linux/v4.10/source/include/sound/soc-dapm.h#L217

However, the reference seems to be on line 213.

Scrolling to the top of the file, I can see that the line numbering isn't aligned with the lines of the file.

Viewing on Google Chrome, Version 56.0.2924.87 (64-bit), on Linux.

Bad Request

Your browser sent a request that this server could not understand.
Apache/2.4.18 (Ubuntu) Server at elixir.bootlin.com Port 443

At least with Safari 11.1.2

browsing through the code I saw the following:

ext = filename[-2:]
if not (ext == '.c' or ext == '.h'): continue

There are a few things that I think could be improved there. First of all, it ignores the cases of extensions: files that end with '.C' or '.H' are ignored. Second, it would be good to have more extensions, such as:

• .S (for assembler)
• .cxx, .c++, etc for C++ files

I would suggest to rsplit() the file name on the '.' character and check if the extension is in a supported list of extensions.

could ofono (https://git.kernel.org/pub/scm/network/ofono/ofono.git) be added to the cross referencer on elixir.bootlin.com?

In assembly files, at least for ARM (I haven't checked other architectures), apostrophes in comments and code cause nearby text to be highlighted as if they were part of a string literal.

An example can be seen here: https://elixir.bootlin.com/linux/v4.19.5/source/arch/arm64/kernel/entry.S#L101

Hi,
Are there plans to expose an API for elixir? (e.g. get_symbol_definition, get_all_references)
If so I'd be willing to contribute.
Thanks!

In projects that have tags that don't follow v3 v3.1 v3.1-rc10 format, the system raises
the following issue. This is observed on 96316cd0b84bfd129f8c5dbc6be77df1a516c780. Works ok on v0.3

A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred.
 /home/naveen/workspace/elixir/http/web.py in <module>()
    129 }
    130 
=>  131 versions = call_query ('versions')
    132 
    133 v = ''
versions undefined, call_query = <function call_query>
 /home/naveen/workspace/elixir/http/web.py in call_query(*args=('versions',))
    108     cwd = os.getcwd()
    109     os.chdir ('..')
=>  110     ret = query.query (*args)
    111     os.chdir (cwd)
    112 
ret undefined, global query = <module 'query' from '/home/naveen/workspace/elixir/http/../query.py'>, query.query = <function query>, args = ('versions',)
 /home/naveen/workspace/elixir/query.py in query(cmd='versions', *args=())
     53         for line in scriptLines ('list-tags', '-h'):
     54             decoded_line = decode(line)
=>   55             topmenu, submenu, tag = decoded_line.split(' ')
     56             if db.vers.exists (tag):
     57                 if topmenu not in versions:
topmenu undefined, submenu undefined, tag undefined, decoded_line = 'Tag_V1', decoded_line.split = <built-in method split of str object>

If i try to search the function msleep via identifier search, i get in Elixir only 4 hits instead of 1627 hits in the old version.

The 'nsmstate' symbol in fs/lockd/clntlock.c is not indexed, though it's not part of a blacklist.
See https://elixir.bootlin.com/linux/v5.4-rc1/source/fs/lockd/clntlock.c#L235

Similarly, in https://elixir.bootlin.com/linux/v5.4-rc1/source/arch/arm/kernel/hw_breakpoint.c,
"dscr" is indexed, but not "didr", even though they are declared in the same way.

Before I might give it a try, looking at script.sh (which adds the 'git' support to elixir, right?), I believe it would be fairly easy to add hg support? Any suggestions?

Hello,
I have a reproductible stacktrace when running elixir on php-src

D: 15100
Traceback (most recent call last):
  File "./update.py", line 141, in <module>
    updateDefinitions (newBlobs)
  File "./update.py", line 84, in updateDefinitions
    line = int (line.decode())
ValueError: invalid literal for int() with base 10: 'function'

As we all know, the correct indentation for tabs in kernel sources is 8, not 4.

One bad side affect is that some previously correctly lined up comments and code are now incorrect when viewing online.

Oh and if anyone ever cuts/pastes from the web view, it will be totally incorrect in their editor :)

Hi sorry to bother (new to linux, apache python etc)

does anybody have a guide for ubuntu 16.04 LTS ?

Problem is installed everything on user Document folder ./script.sh and update.py and they work
database is created (at least I can see the files) but Apache doesnt find the web.py. Tried a lot of configuration but no luck.

Kind of Ubuntu tries to store webpages in var/www/html tried that way but wasnt able even to set env path so no way to run script.sh.

Any hint suggestion ? thanks P.

I tried the docker image on my code repo other than Linux kernel repository.

I enabled debug in web.py and output the error log to /tmp folder.

import cgitb
cgitb.enable(display=0, logdir='/tmp')

Here is the error I got from html page /tmp/tmp32xjo7k5.html:

...
<!-- The above is a description of an error in a Python program, formatted
     for a Web browser because the 'cgitb' module was enabled.  In case you
     are not reading this in a Web browser, here is the original traceback:

Traceback (most recent call last):
  File "/usr/local/elixir/http/web.py", line 138, in &lt;module&gt;
    lines = do_query ('versions')
  File "/usr/local/elixir/http/web.py", line 109, in do_query
    a = query.query (*args)
  File "/usr/local/elixir/http/../query.py", line 44, in query
    if db.vers.exists (p.split(b' ')[2]):
IndexError: list index out of range

-->

hi,
$ ./query.py command is very good, but when I use the chrome browser, It error

Hi

First of all, I'm completely not aware of Apache server configurations, so this might be a simple PEBKAC. I did follow all steps of the installation manual. My server (httpd) is up and running, however I'm getting an error when it want to run web.py, since there's no SCRIPT_URL variable set. What should be the proper value for this variable? Should I set it or it should be somehow set automatically by httpd? Below is the tree of the test linux repository I'm trying to run. I've only modified elixir scripts a little, to return just 2 of the tags instead of all of them. Rest is unchanged.

(all Linux data is inside the "repo" directory - this is wrongly presented here)
user@server:~/data/proj_elixir$ tree -L 3
.
└── linux
├── data
│   ├── blobs.db
│   ├── definitions.db
│   ├── filenames.db
│   ├── hashes.db
│   ├── references.db
│   ├── variables.db
│   └── versions.db
└── repo
├── arch
├── block
├── certs
├── COPYING
├── CREDITS
├── crypto
├── Documentation
├── drivers
├── firmware
├── fs
├── include
├── init
├── ipc
├── Kbuild
├── Kconfig
├── kernel
├── lib
├── MAINTAINERS
├── Makefile
├── mm
├── net
├── README
├── samples
├── scripts
├── security
├── sound
├── tools
├── usr
└── virt

I came along a source file where I would like to refer to the beginning of the function “write_regs” (for a corresponding software development discussion).
It seems that the following links would be appropriate in principle at the moment.

• https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/media/usb/go7007/s2250-board.c?h=v4.14-rc1#n302
• http://elixir.free-electrons.com/linux/v4.14-rc1/source/drivers/media/usb/go7007/s2250-board.c#L298

Now I wonder about the shown line numbers. How could they be different when they should refer to the same source code place according to the used version tag?

In kernel/trace/trace_events.c in Linux, around 1300 lines are not properly indexed.

The problem starts here: https://elixir.bootlin.com/linux/v4.17/source/kernel/trace/trace_events.c#L2149
and seems to fix itself about 1300 lines later: https://elixir.bootlin.com/linux/v4.17/source/kernel/trace/trace_events.c#L3412

I strongly suspect the double quote (if (*ptr == '"') {) in the initial line to be the culprit, probably confusing the parser what's inside a string and what's not - this also causes some strings inside the faulty range to have links in them (e.g.,
https://elixir.bootlin.com/linux/v4.17/source/kernel/trace/trace_events.c#L3159)

Looking at the definitions of "jiffies" in https://elixir.bootlin.com/linux/v5.3.5/source/arch/arc/kernel/vmlinux.lds.S#L17 , we can see some assignments improperly considered as definitions.

There is probably a need for special processing of the .S files to avoid this.

Recently, website(https://elixir.bootlin.com) is very slow. Jumping to the function need more than 10 seconds sometimes. Confused with this. Is there some bugs with xlixir or too many persons access your website?
Feel sorry to report this issue here, but I do not know where to report this isssue.
Hope to receive your response.
Best Regards.

Usually searching some kernel config as identifier doesn't work, so "https://elixir.bootlin.com/linux/latest/ident/CONFIG_SCHEDSTATS" returns empty page.

But today I found it works with some kernel config like "https://elixir.bootlin.com/linux/latest/ident/CONFIG_PARAVIRT" and I think the reason is that it has something like "#undef CONFIG_PARAVIRT" in the code.

Then can we make it available generally for "#ifdef xxx"?
Use case: As mentioned, we can check the code very easily which is dependent to a certain kernel config in Linux kernel source tree.

In Linux 4.9, the "vb2_queue" identifier should be recognized in include/media/videobuf2-core.h both as a struct (line 493) and as a struct member (line 242), but only the later is actually reported.

I am frequent user of elixir and one issue that I encountered is while searching for an identifier if I put a space before the identifier's name in Search Identifier box, it does not show any results.
If I search the same identifier without a space before the name, I get results.
So, while searching for the identifier in the db, spaces before the identifier should be removed first.

I can create a PR proposing this fix.

Searching without space: string == "fpga_bridge_ops"

Searching with space: string == " fpga_bridge_ops"

It seems that something went wrong when creating the database of PHP, because elixir produces non-working pages:

==> /var/log/apache2/error.log <==
[Tue Jun 27 08:42:12.979348 2017] [http:error] [pid 11443:tid 116669335639808] [client 192.168.135.96:60244] AH02429: Response header name '<!--' contains invalid characters, aborting request, referer: http://10.1.76.1/php/php-7.1.6/ident/zend_string

==> /var/log/apache2/other_vhosts_access.log <==
elixir.example.com:80 192.168.135.96 - - [27/Jun/2017:08:42:12 +0000] "GET /php/php-7.1.6/source/Zend/zend_API.c HTTP/1.1" 500 777 "http://10.1.76.1/php/php-7.1.6/ident/zend_string" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/58.0.3029.110 Chrome/58.0.3029.110 Safari/537.36"

Here is my Apache2 configuration:

<Directory /data/git/elixir/http/>
    Options +ExecCGI
    AllowOverride None
    Require all granted
    SetEnv PYTHONIOENCODING utf-8
    SetEnv LXR_PROJ_DIR /data/git/elixir-data
</Directory>

AddHandler cgi-script .py

<VirtualHost *:80>
    ServerName elixir.example.com
    DocumentRoot /data/git/elixir/http

    RewriteEngine on
    RewriteRule "^/$" "/php/latest/source" [R]
    RewriteRule "^/.*/(source|ident|search)" "/web.py" [PT]
</VirtualHost>

I would like to look at implementations of functions which output diagnostic information. Now I observe that a search by your evolving cross reference tool for a function like “dev_notice” points the header file “device.h” out. I would expect that another source file which contains corresponding implementations will be shown.

How are the chances that the software situation can be improved further similar to information in the previous bug report “Identifier search doesn't find all function occurrences”?

A suggestion from Bram Vlerick [email protected]...

Are there any plans to make the search identifier do some kind of autocomplete/fuzzy search?

I have built Elixir docker using docker/Dockerfile

When running it, and trying to access localhost:8080 apache reports an internal server error

In apache error.log it is detailed:
AH02429: Response header name '<!--' contains invalid characters, aborting request

in the docker container at apache.conf there is:
-e
HttpProtocolOptions Unsafe

it seems as HttpProtocolOptions Unsafe it is not solving it.

Apache version in the docker image is:
apachectl -v
Server version: Apache/2.4.25 (Debian)
Server built: 2018-11-03T18:46:19

Hi there!
Thanks for your hard work :)

Unfortunately it seems that on my system (latest linux, lastest chromium browser)
the line numbering does not fit the contents...

As you can see here, line 116 should be struct bus_type {

When I can be of any help, please tell me!

Reported by Jan Sokolowski (thanks!)

I've got a question, as a person who often has to browse linux kernel
elixir when debugging a kernel: Is it possible to add function
pointers in structs as symbols to search for?

For example: in net_device_ops:
https://elixir.bootlin.com/linux/v5.1-rc4/source/include/linux/netdevice.h#L1259
I'd like to know where ndo_init is called throughout the kernel by
clicking on it.

Is it possible to add this feature?