coatisoftware / sourcetraildb Goto Github PK

How to read .srctrldb？Is there a command to get the internal data

Hi, thank you for open sourcing a library to interact with the db

I'm having trouble with the NameHierarchy structure.

Right now, to keep track of class -> method relationships, I have to track (and update!) with a separate data structure every NameHierarchy so that they will show up nicely grouped in Sourcetrail.

Wouldn't it be easier to allow developers to manage this use case with References (that is creating an edge) so that the resulting hierarchy will be implicit?

Even in https://github.com/CoatiSoftware/SourcetrailPythonIndexer/blob/master/indexer.py#L394 the NameHierarchy related code makes up 1/8 th of the code base.

With other languages it gets even worse!

I can't see any advantage with the current api pushing the complexity onto every new Language Backend rather than taking the toll in the frontend

I would be interested in adding typescript support to SourcetrailDB. This would also lay the groundwork for JavaScript support as well.

Specifically, I would like to ensure that react code is supported as well.

I think it would be lovely if we could visualize big buildsystems of CMake (e.g.) with Sourcetrail. So some language bindings for that would be pretty neat as well.

49e8458#diff-e265df4e0fdad58edad87694c8140a4cR365

Hello !

I've started the work concerning the rust bindings of SourcetrailDB. It is a (very) early (and non yet working) version of those, made with rust-bindgen.

You can see the progress and contribute at https://github.com/Champii/SourcetrailDB/tree/master/resources_rust

The current approach is to use rust-bindgen directly on the generated static library, but that may be inefficient as rust-bindgen does not handle std types well (std::string for example). Their doc recommend to generate them all as opaque types, but then we lose the ability to use them.

One of the possible workaround is to create a C ABI to interface the C++ code with Rust, or to try the method used by the python bindings and wrap the most commonly used features in a lightweight and thus Rust-friendly C++ library.

I'll try them both and consider the one that fit the most.

Feel free to share your point of view on the matter, and/or throw some suggestions on the best way to handle that !

If I can be of any help, I'd be glad to :)

As already mentioned in issue #2, the EdgeKind::AGGREGATION is only created by Sourcetrail dynamically and is never stored to the databse.

Are there any plans to add bindings for .NET languages?

If I understand correctly, this could easily be done via SWIG, which can output C# as a target language. The resulting C# might then be compiled into a DLL together with any other dependencies and packaged as a NuGet package; the bindings would be available to any .NET developer.

This would open many possibilities:

• C# / VB.NET indexing, via Roslyn
• F#, using the F# compiler
• Direct IL indexing

So I'm trying to build the project on Ubuntu for using it with SourcetrailPythonIndexer
but I have a problem:

$ cat /etc/issue
Ubuntu 18.10 \n \l
$ g++ --version
g++ (Ubuntu 8.2.0-7ubuntu1) 8.2.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ cmake --version
cmake version 3.12.1

CMake suite maintained and supported by Kitware (kitware.com/cmake).

$ swig -version

SWIG Version 3.0.12

Compiled with g++ [x86_64-pc-linux-gnu]

Configured options: +pcre

Please see http://www.swig.org for reporting bugs and further information

The exact error:

$ cmake -DBUILD_BINDINGS_PYTHON=ON ..
fatal: No names found, cannot describe anything.
-- SourcetrailDB Version: v.db.p
-- Interface Version: 
-- Database Version: 
-- Patch Version: 
-- The SourcetrailDB Python bindings will be built.
-- Found SWIG: /usr/bin/swig3.0 (found version "3.0.12") 
-- The examples will be built.
-- Configuring done
-- Generating done
-- Build files have been written to: /home/bender/SourcetrailDB/build

$ make lib_core
[  7%] Building CXX object core/CMakeFiles/lib_core.dir/src/DatabaseStorage.cpp.o
/home/bender/SourcetrailDB/core/src/DatabaseStorage.cpp: In static member function ‘static int sourcetrail::DatabaseStorage::getSupportedDatabaseVersion()’:
/home/bender/SourcetrailDB/core/src/DatabaseStorage.cpp:35:3: error: return-statement with no value, in function returning ‘int’ [-fpermissive]
   return DATABASE_VERSION;
   ^~~~~~
make[3]: *** [core/CMakeFiles/lib_core.dir/build.make:89: core/CMakeFiles/lib_core.dir/src/DatabaseStorage.cpp.o] Error 1
make[2]: *** [CMakeFiles/Makefile2:93: core/CMakeFiles/lib_core.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:105: core/CMakeFiles/lib_core.dir/rule] Error 2
make: *** [Makefile:118: lib_core] Error 2

Would be awesome to have support for Swift.

I'm available to help you out!

This is basically a followup from 421

Sorry, my English is not very good. I use Google Translate to translate Chinese into English, and it may not be clear. Here are my questions:
I used Sqllite to open the srctrldb file and found two data tables, node and edge. They contain the "type" field. I would like to ask, what does 1,4,8,512 in the edge table mean? And what does the content of the type field in the node table mean?

For example: 49e8458#diff-e265df4e0fdad58edad87694c8140a4cR365

In my TypeScript/Node implementation, I corrected it in the publicly-facing signature, but it of course still shows up in the C++ call: https://github.com/LouisStAmour/sourcetraildb/blob/master/Writer.cc#L317

Language Server Protocol provides calls for many (possibly all) of the language analysis that Sourcetrail needs. There are now open source langservers for many different languages - sometimes multiple implementations for a single language.

It's possible that configuring Sourcetrail to use an existing language server could reduce the work of supporting a language down to a simple config file.

EDIT: If you're interested in general discussion about LSP and/or LSIF support in Sourcetrail (which would make this issue redundant), that's happening here: CoatiSoftware/Sourcetrail#685 . This issue is about implementing that support as an external framework usable by bindings.

This tool would be very useful to visualize JavaScript dependencies.

Webpack has built-in tools that can do this to some degree (See "Webpack Analyzer" at this article), but the resulting visualization is not pleasant to use as the amount of code grows. Some issues in webpack's analyzer include - too many nodes & lines; "filtering down" could be somewhat better (filtered-out nodes remain in graph, but less opaque); performance suffers due to it being an interactive force-directed graph with very many elements that is implemented with JS & canvas, etc.

The shortest/simplest path to having SourceTrail be able to show JS dependencies - I think - might be to create a script (a "binding" ?) that simply imports the "webpack-stats.json", converts to data format/structure needed internally for SourceTrail, and then allows to visualize.

After making issue, realized an existing issue #2 exists for TypeScript support. As well as in other repo - CoatiSoftware/Sourcetrail#294. If there's more progress with that/those approaches, might be worth closing this topic.

Will the Java binding SourcetrailDB.jar be in maven repository ?

Crystal is inspired by Ruby so I guess they might share some logic in a custom indexer.

When properties or methods have special characters or diacritics (º, ª, á, ñ, ç, etc.) I get the following result:

Unable to deserialize name hierarchy "{ "name_delimiter" : ".", "name_elements" : [ { "prefix" : "", "name" : "ExampleNamespace", "postfix" : "" },
{ "prefix" : "", "name" : "ExampleClass", "postfix" : "" },
{ "prefix" : "public string", "name" : "Peça", "postfix" : "" }] }": [json.exception.parse_error.101] parse error at 263: syntax error - invalid string: ill-formed UTF-8 byte; last read: '"Peça'

Here's the formatted json:

{
    "name_delimiter": ".",
    "name_elements": [
        {
            "prefix": "",
            "name": "ExampleNamespace",
            "postfix": ""
        },
        {
            "prefix": "",
            "name": "ExampleClass",
            "postfix": ""
        },
        {
            "prefix": "public string",
            "name": "Peça",
            "postfix": ""
        }
    ]
}

This is an example that generates the above json and error (C#):

namespace ExampleNamespace
{
    public class ExampleClass
    {
        public string Peça { get; set; }
    }
}

Fortran may sound a bit outdated but it is still widely used in simulation software as it is often a more natural fit for the application experts.

I personally came across many academic projects that consist of a major part of Fortran code.
A few notable example projects are:

• SeisSol
• PFLOTRAN
• CalculiX
• FASTEST

Fortran support would definitely help to speed up students and researchers to find their way inside such a code base and thus increase the maintainability of valuable research software.

Problem Description
The graph-view gets cluttered very quickly with non-indexed symbols.

This is especially problematic in C++ while using caller/callee graphs with multiple levels.
In such cases, the view gets quickly filled with the non-indexed interface-symbols of other libraries such as the STL, Boost and other dependencies.

Proposed Solution
I propose a local and global way of hiding non-indexed symbols.

1. Add an option to the context menu of the graph view allowing to "Hide all non-indexed nodes".
2. Add an option in the preferences to globally disable non-indexed symbols in the graph view "Show non-indexed symbols in graph".
   Possibly next to "Show build-in types in graph when referenced".

Thank you very much for developing this awesome software!

System integrations are frequently implemented as shell scripts and PowerShell is getting more users every year (in 2020 progressed to rank nr 7 of most used languages on github).
One of the benefits relevant here is its build-in AST support, explained in Tobias Weltner's expert blog here

System integration requires a different domain knowledge than software development and currently lacks specific tools support, hence making Sourcetrail available would be a great improvement.

I'm a system integrator, hence can't really judge if the requirements defined in Sourcetrail language extension guide
are already met by PowerShell build-in AST support

Could you please step in here, so we can get an idea of required effort to get it going?

I have created the Perl sourcetraildb module using SWIG. PPI can be used to parse the Perl AST.