log2timeline / l2tdevtools Goto Github PK

in review.py :

self._git_repo_url = b'https://github.com/log2timeline/{0:s}.git'.format(
self._project_name)
If the repo is cloned using:
git clone https://github.com/log2timeline/plaso instead of git clone https://github.com/log2timeline/plaso.git

then the origin url will mismatch in

origin_git_repo_url = self.GetRemoteOrigin()                                 
return origin_git_repo_url == self._git_repo_url

because of the trailing ".git" in self._git_repo_url.

• objgraph - https://mg.pov.lt/objgraph/
• xdot - https://pypi.python.org/pypi/xdot

• add support for unicode characters in description

Improvements to dpkg build target:

• fix binplist existing package test
• dpkg/rules set x bit
• check binplist dpkg files and rename to python-binplist (needs project move)
• docopt fix dpkg "None: no @ found in email address part."
• requests handle v#.#.#.tar.gz file being created
• missing python-setuptools build dependency not detected
  • http://codereview.appspot.com/250420043
• python-sqlite
  • Python.h: No such file or directory - add python-dev as a build dependency
  • sqlite3.h: No such file or directory - add libsqlite3-dev as a build dependency
• add pytsk support, what about package version number?
• psutil
  • Python.h: No such file or directory - add python-dev as a build dependency
  • issues with 2.2.1 breaking on tests
• support different dpkg files for different debian/ubuntu versions
  • replace python-dev by python-all-dev
  • detect if the dpkg files contain python 3.4 (not supported by precise) and fallback to a python 2 only build?
• enforce dpkg_name above python- expansion for pylint
• rebuild of: acora
• add python3 support for dpkg_build_dependencies
  • https://codereview.appspot.com/320850043/
• handle non 1 dpkg versions in clean up
  • https://codereview.appspot.com/323870043
• move to source format 3.0
  • repackage zip a tar.gz
  • clean before build for second distribution
• l2tdevtools clean up python3 deb files
• dpkg generate warning if maintainer is missing
  • moved to #571
• check and fix dpkg dependencies (continue with gevent)
• add option to force regeneration of dpkg files ?
  • moved to #570
• determine setup.py dependencies based install_requires or requires
  • moved to #544

I would rather have just space delimited package names, so I can copy paste the output for apt-get install

Have:

$ ./tools/build.py --projects=libscca dpkg
[INFO] Processing: libscca
[WARNING] Missing build dependencies: quilt, python3-setuptools, python3-all, byacc, python-all, python-all-dev, python3-all-dev.
Failed building: libscca
 
Failed buiding:
        libscca

Would prefer:

$ ./tools/build.py --projects=libscca dpkg
[INFO] Processing: libscca
[WARNING] Missing build dependencies: quilt python3-setuptools python3-all byacc python-all python-all-dev python3-all-dev
Failed building: libscca
 
Failed buiding:
        libscca

• add support for presets
• handle platform specific package names
• Have update script detect python version
• split l2tbinaries in version 2 and 3 binaries

Currently the google code package names are ignored on on Mac OS X, this causes multiple versions of the same package to be installed.

Add srpm and copr build targets to support
https://copr.fedorainfracloud.org/groups/g/gift/coprs/

Have a look at Microsoft Visual C++ Compiler for Python 2.7 to replace full fledged VS 2008 express

http://www.microsoft.com/en-us/download/details.aspx?id=44266

Reported in: log2timeline/plaso#705

Running review.py test and review.py lint requires connection to github. Would be nice run these tests while not connected online and with no upstream remote available. (ie. run tests and linting based off of a diff of the master to feature branch)

Change the merge_submit to double check name and email address and determine why it does not find the name in certain situations.

• Implement fix: https://codereview.appspot.com/259670043/
• Update dfvfs scripts: http://codereview.appspot.com/262320043
• Update l2tdevtools scripts: http://codereview.appspot.com/261150043

Add a packaged version of requests

• GIFT
• macosx
• win32
• win64

Update documentation:

• ubuntu
• fedora
• windows
• macosx

• handle github releases archive tags with a prefix e.g. psutil
• http://codereview.appspot.com/218630043
  • Indicate project type in configuration e.g. libyal, python, etc. or detect installer configure, setup.py, etc.
  • allow to disable per build target
• add plaso as a project
• add project groups e.g. libyal, plaso, grr - presets
• automate packing libtsk and pytsk3, manual packaging error prone
  • fixed by stand-alone pytsk package
• add support to build.py to only build for python 2.7
• Handle zip only source packages e.g. WTForms, simplegeneric on pypi
  • issue in download helper
• change download helper to use configured URL information e.g. pretty-yaml
  • deprecate download_name, detect this from download_url instead
• add Python 3 support e.g. rpm prefix with python3
  • generate spec file similar to: https://fedoraproject.org/wiki/Packaging:Python
  • add support for Python 2 only targers e.g. lzma backports
• add support for new libyal setup.py
  • https://github.com/joachimmetz/l2tdevtools/tree/libyal
• rename source_filename to source_package_filename
• check build dependencies before build
  • output missing dependencies as user output; instead of ending up in logs as "Unmet ..."
  • #535
• check discrepancy of build dependencies and dependencies between dpkg files and projects.ini
• provide build summary
  • failed builds
  • undefined packages
  • new builds
• handle versions
  • add support for rc suffix https://pypi.python.org/pypi/M2Crypto
  • check if version is above minimum or below maximum
  • if exact version is provided notify if newer available version is found? e.g. sleuthkit
  • fix clean up removing a-b-* when cleaning up a-*
• selective builds
  • check package names case insensitive
• semi-automatically determine build dependencies
• determine build older based on dependencies?
• add libcfile + support for other libc and libf libraries
• add MSYS-MinGW build option
• add build from git support (partial)
• Improve version number support based on https://www.python.org/dev/peps/pep-0440/
• add option to control python binding build versus DLL build
• add build source target for docker/virtenv
• Add option to toggle package signing (dpkg and dpkg-source)
• add patch support
  • msi
  • rpm
  • dpkg, dpkg-source (reapply provided patches with quilt)
  • pkg
• do not remove Flask-Sub source packages on clean
• add pypi build target
  • have a wheel build target instead
• At start add build status e.g.:

Build target: rpm
Build directory: MyBuild/
Configuration file: MyPath/projects.ini

To do:

• Make code more modular (improve re-use)
• add tests
• change update and check script to use config file

To do:

• fix binplist dpkg files and rename to python-binplist
• fix build of python-sqlite - moved to #4

Close dependency script issues reported under plaso

• log2timeline/plaso#82
• log2timeline/plaso#96
• log2timeline/plaso#97

improve stats.py to:

• list outstanding code reviews
• list historical code reviews
• list historical commits
• add description of commits
• list outstanding issues
• list historical issues
• list outstanding PRs
• list historical PRs
• add weekly stats, maybe support time ranges?

Currenly there are issues with the following packages in dev:

• pytz
  • Python 2 package name not prefixed
• pyparsing
  • Python 2 package name prefixed with python2
  • conflicting meta package removed from dev for now

Error: package pyparsing-2.1.10-1.fc25.noarch requires python-pyparsing = 2.1.10-1.fc25, but none of the providers can be installed

• python-zmq
  • Python 2 package name prefixed with python2
  • conflicting meta package removed from dev for now

Error: package python-ipython-console-3.2.1-8.fc25.noarch requires python-zmq, but none of the providers can be installed

• python3-requests

  • some dependencies are bundled into source, should provide them separately
  • removed from dev for now

• dtfabric

Reported by @Onager

Setting up python-astroid (1.4.4-2ppa1~wily) ...
  File "/usr/lib/python2.7/dist-packages/astroid/tests/testdata/python3/data/module2.py", line 100
    print('bonjour', file=stream)
                         ^
SyntaxError: invalid syntax

dpkg: error processing package python-astroid (--configure):
 subprocess installed post-installation script returned error exit status 101
dpkg: dependency problems prevent configuration of python-pylint:
 python-pylint depends on python-astroid; however:
  Package python-astroid is not configured yet.

Work-around

.PHONY: override_dh_python2
override_dh_python2:
    dh_python2 -V 2.7 setup.py

• changes for project site move
• add isort dependency
• add mccabe dependency
  • pytest-runner dependency?
• Fix #63
• add backports.functools-lru-cache dependency
• fix binary package name of pylint
• add setuptools_scm dependency (for Trusty)
  • add python setup.py egg_info to override_dh_auto_install
  • patch get_version() in __init__.py to contain hardcoded version
• fix setuptools Python3 build
  • fix Python3 build dependencies
• mccabe failing to build on xenial

https://fedoraproject.org/wiki/Changes/ReplaceYumWithDNF

• add python guppy for debugging
  • https://pypi.python.org/pypi/guppy/
• add efilter dependencies dateutil, pytz, six

dpkg_dependencies: python-dateutil,python-six,python-tz

look into review nobrowser not working on update

It's been moved now: https://github.com/erocarrera/pefile

• prefix python packages with python- - not needed
• add sleuthkit support - no longer needed
• add pytsk3 support
• check for dependencies

Ref log2timeline/plaso#193

• Win32 build
• Win64 build
  • libsodium compile errors VS2010 (POSIX type replacements)
  • zeromq compile errors VS2010 (POSIX type replacements)
  • alternative is to use one of the official releases https://github.com/zeromq/pyzmq/releases
• Mac OS X build
• Ubuntu build - Already provided by Ubuntu too much packages dependent on ZeroMQ to cleanly distribute via GIFT
• Travis CI - Ubuntu precise (backport 14 from trusty to precise)
  • build cython architecture independent with --no-cython-compile, removed python-cffi dependency
  • need amd64 build

• Fix git pull issue on close
• fix version not being updated on merge
• Check why the rietveld close issue is still not working
  • missing cookie jar - use OAuth instead
• change/remove missing test file script
• Move review scripts to Python - to support Windows
  • update l2tdevtools script with plaso scripts
  • fix limitation in merge script that CL description cannot contain a ':'
  • add a submitted comment to code review
  • change http:// to https:// where possible
• fix issue argparse yielding different results when run from review.py
• review.py add functionality to "test"
• review.py add option for unattended mode - do not ask for confirmation
• review.py create - do not generate '[project_name]' in PR description
• document usage of BROWSER environment variable
• review.py "lint" add option to lint all files
• update dfwinreg scripts
  • removed shell scripts
• update dfvfs scripts
  • removed shell scripts
• update plaso scripts
  • removed shell scripts
• Provide pylint in l2tbinaries
• https://codereview.appspot.com/309000043/
  • create - check if review file exists first
  • close - notify if review file is missing or codereview issue cannot be closed
  • close - support user:branch as input besides branch
  • added update-authors functionality
• add lint_and_test option
• fix issue with create --nobrowser
  • Create aborted - unable to create codereview issue.
  • Issue created message not at the start of the line

Remaining items moved to: https://github.com/log2timeline/l2treviewtools/issues/22

• add tool to generate dpkg files
• Update build instructions to use the script
• Remove deprecated dpkg files

Step1: https://codereview.appspot.com/240130043

• add M2Crypto for GRR
• add wcwidth for GRR
• add acora, codegen, distorm3, gevent, gevent-websocket, intervaltree, pycrypto, pyelftools, Flask, Flask-Sockets, PyAFF4 for rekall
• add Werkzeug for Flask
• add Sphinx for Werkzeug
• add alabaster, Babel, docutils, Jinja2, MarkupSafe, nose, Pygments, snowballstemmer, sphinx_rtd_theme for Sphinx
• add MarkupSafe for Jinja2

requests Makefile triggers:

pip: Command not found

force setup.py as build system

Fix jinja2 no package content - requires explicit install definition in rules

.PHONY: override_dh_auto_install
override_dh_auto_install:
    mkdir -p debian/python-jinja2/usr/lib/python2.7/site-packages
    PYTHONPATH=debian/python-jinja2/usr/lib/python2.7/site-packages python ./setup.py install --prefix debian/python-jinja2/usr
    mv debian/python-jinja2/usr/lib/python2.7/site-packages debian/python-jinja2/usr/lib/python2.7/dist-packages
    rm -f  debian/python-jinja2/usr/lib/python2.7/dist-packages/easy-install.*
    rm -f  debian/python-jinja2/usr/lib/python2.7/dist-packages/site.*
    mv debian/python-jinja2/usr/lib/python2.7/dist-packages/Jinja2*.egg/jinja2 debian/python-jinja2/usr/lib/python2.7/dist-packages

To do

• Werkzeug not building on launchpad (not that critical since it seems to be already packaged)
• handle distorm3 not providing a tar.gz source package but a zip instead
• psutil version 2?
• django 1.6
• selenium
• rekall

add l2tbinaries tests

• Add tests
• Complete VS 2010 reader
• Complete VS 2012 reader
• Complete VS 2013 reader
• Complete VS 2013 writer
• Create a msbuild reader and writer?

• rewrite history to fix email addresses of committers

• add option to compare against tsv
• handle dpkg and rpm packaging number
• add pypi support
• add osc support
  • moved to #686

• update libfvde and do a new release; some fixes for dpkg files need to be included
• upload libfvde to GIFT

Generate the "ontology data types" based on the source

Noticed with issue https://codereview.appspot.com/281870043/

• requires xz sources: http://tukaani.org/xz/
  • build: xz_win.sln
  • code C99 or later does not work with VS2008
• requires: https://pypi.python.org/pypi/backports.lzma/0.0.3
  • set up paths to build of xz sources
• tox need xz-headers to work

TODO:

• add dpkg support
• add osc support
• add MSI
• add PKG
• add support for pyliblzma and bz2 download file
• http://tukaani.org/xz/

create a script to check if gift and l2tbinaries are up to date

• for gift stable check if
  • stable in sync with dev
  • dev in sync with testing
  • testing in sync with build directory
• for l2tbinaries check if
  • osx in sync with build directory
  • win32 in sync with build directory
  • win64 in sync with build directory

The MSIs get installed in C:\Miniconda-x64\Lib\site-packages instead of C:\Python27\Lib\site-packages

• rename Python packages to python
  • overwrite bdist_rpm target ? only need the package name to change
• handle artifacts data files

The page_content of http://www.zlib.net currently has no line matching

u'<A HREF="http://zlib.net/{0:s}-([0-9].[0-9].[0-9]).tar.gz"

the link is relative:
<A HREF="zlib-1.2.11.tar.gz"

Add script to maintain l2tbinaries:

• remove older versions
• add newer versions
• update SHA256SUMS
• clean up git history (+/-)

http://pythonwheels.com/

• code clean up
• remove Ubuntu update in favor of gift PPA
• move to l2tbinaries repo instead of Google Drive folder.
• allow to specify list of packages to update
  • do not download other packages
• do not remove local copies of already downloaded packages (OSX)
• add download only option

Improvements to msi build target:

• fix rebuild of existing coverage - has -py2.7 suffix
• vs2010 build defaults to vs2008 on the first run?
• VS 2015 support
• set up zlib source
• fix rebuild of existing pefile (version number breaks build_msi only supports x.y.z)
  • solved by patch support
  • upstream fix still pending
• fix missing build dependency libcrypto
• add support for libyal new setup.py
  • Update documentation
• fix dateutil (re)build - something to do with setuptools version?
  • https://codereview.appspot.com/341240043/
• Make changes to stop GNU patch.exe to have admin access - patch currently not used
• check if build is ignored if package file already exists
  • libyal - msi broken due to package naming changes
• Handle libyal libraries without python-bindings
  • detect if --enable-python or setup.py is present
• set up dokan source
• coverage not arch dependent, check if this breaks on other platforms (or is this platform dependent?)
• add pywin32 support (partial)
  • check for mc.exe
  • missing headers for certain features
• Generate wix MSI files instead of relying on setup.py bdist_msi?
• Add bdist_wininst support?
  • Add UAC support ? ./setup.py --user-access-control=auto bdist_wininst, also see: https://docs.python.org/2/distutils/builtdist.html

Notes on pywin32 support:
error: command 'mc.exe' failed: No such file or directory

• mc.exe is not part of VS 2008 express, install the Win7 SDK (part of the tools installation option). Make sure it is in PATH.

cl : Command line warning D9024 : unrecognized source file type '3.0',
fatal error C1083: Cannot open source file: '3.0': No such file or directory

• some internal cl.exe error seen in combination with VS2008 and Win7 SDK

Some ideas for enhancements

It looks like download urls have changed, and include some unexpected elements:
https://pypi.python.org/packages/d3/8b/a82129243571dba8f37e8b488265984d86ddd881faf82745ede600627871/pylint-1.5.5.tar.gz#md5=208a6365f81ba072293539ae51614d09

• remove old versions of source packages
• spec make sure to include python-devel and python3-devel if needed
• overwrite description
• detect and add license file
• detect and add documentation files
• spec file include python 3 dependencies and move python 2 dependencies to correct packages
• handle scrips/tools
  • pbr, ipython, artifacts (also /usr/share), XlsxWriter
• support different source extraction path or repackage source package?
  • psutil, efilter, astroid,

Note: Issues are disabled in the l2tbinaries repo. So I am adding it here.

The msi installer files created in the l2tbinaries repo does not allow you to change the install location from command line. The TARGETDIRX property allows you to change the path, but there is no way to set it to use that path instead of the path found in the Window's registry.

I am trying to create a script that automatically installs all the windows binary files found in the l2tbinaries repository into a virtualenv.