Git Product home page Git Product logo

dlltracer-python's Introduction

dlltracer

The dlltracer tool is an assistive tool for diagnosing import errors in CPython when they are caused by DLL resolution failures on Windows.

In general, any DLL load error is reported as an ImportError of the top-level extension module. No more specific information is available for CPython to display, which can make it difficult to diagnose.

This tool uses not-quite-documented performance events to report on the intermediate steps of importing an extension module. These events are undocumented and unsupported, so the format has been inferred by example and may change, but until it does it will report on the loads that actually occur. However, because it can't report on loads that never occur, you'll still need to do some work to diagnose the root cause of the failure.

The most useful static analysis tool is dumpbin, which is included with Visual Studio. When passed a DLL or PYD file and the /imports option, it will list all dependencies that should be loaded. It shows them by name, that is, before path resolution occurs.

dlltracer performs dynamic analysis, which shows the DLLs that are loaded at runtime with their full paths. Combined with understanding the dependency graph of your module, it is easier to diagnose why the overall import fails.

Install

pip install dlltracer

Where the pip command may be replaced by a more appropriate command for your environment, such as python -m pip or pip3.9.

Use

Note: Regardless of how output is collected, this tool must be run as Administrator. Otherwise, starting a trace will fail with a PermissionError. Only one thread may be tracing across your entire machine. Because the state of traces is not well managed by Windows, this tool will attempt to stop any other running traces.

A basic trace that prints messages to standard output is:

import dlltracer
import sys

with dlltracer.Trace(out=sys.stdout):
    import module_to_trace

The output may look like this (for import ssl):

LoadLibrary \Device\HarddiskVolume3\Windows\System32\kernel.appcore.dll
LoadLibrary \Device\HarddiskVolume3\Program Files\Python39\DLLs\_ssl.pyd
LoadLibrary \Device\HarddiskVolume3\Windows\System32\crypt32.dll
LoadLibrary \Device\HarddiskVolume3\Program Files\Python39\DLLs\libcrypto-1_1.dll
LoadLibrary \Device\HarddiskVolume3\Program Files\Python39\DLLs\libssl-1_1.dll
LoadLibrary \Device\HarddiskVolume3\Windows\System32\user32.dll
LoadLibrary \Device\HarddiskVolume3\Windows\System32\win32u.dll
LoadLibrary \Device\HarddiskVolume3\Windows\System32\gdi32.dll
LoadLibrary \Device\HarddiskVolume3\Windows\System32\gdi32full.dll
LoadLibrary \Device\HarddiskVolume3\Windows\System32\msvcp_win.dll
LoadLibrary \Device\HarddiskVolume3\Windows\System32\imm32.dll
LoadLibrary \Device\HarddiskVolume3\Program Files\Python39\DLLs\_socket.pyd
LoadLibrary \Device\HarddiskVolume3\Program Files\Python39\DLLs\select.pyd

A failed import may look like this (for import ssl but with libcrypto-1_1.dll missing):

LoadLibrary \Device\HarddiskVolume3\Windows\System32\kernel.appcore.dll
LoadLibrary \Device\HarddiskVolume3\Program Files\Python39\DLLs\_ssl.pyd
LoadLibrary \Device\HarddiskVolume3\Windows\System32\crypt32.dll
LoadLibrary \Device\HarddiskVolume3\Program Files\Python39\DLLs\libssl-1_1.dll
Failed \Device\HarddiskVolume3\Windows\System32\crypt32.dll
Failed \Device\HarddiskVolume3\Program Files\Python39\DLLs\libssl-1_1.dll
Failed \Device\HarddiskVolume3\Program Files\Python39\DLLs\_ssl.pyd
Traceback (most recent call last):
  File "C:\Projects\test-script.py", line 28, in <module>
    import ssl
  File "C:\Program Files\Python39\lib\ssl.py", line 98, in <module>
    import _ssl             # if we can't import it, let the error propagate
ImportError: DLL load failed while importing _ssl: The specified module could not be found.

Notice that the missing DLL is never mentioned, and so human analysis is necessary to diagnose the root cause.

Write to file

To write output to a file-like object (anything that can be passed to the file= argument of print), pass it as the out= argument of Trace.

import dlltracer

with open("log.txt", "w") as log:
    with dlltracer.Trace(out=log):
        import module_to_trace

Collect to list

To collect events to an iterable object, pass collect=True to Trace and bind the context manager. The result will be a list containing event objects, typically dlltracer.LoadEvent and dlltracer.LoadFailedEvent.

import dlltracer

with dlltracer.Trace(collect=True) as events:
    try:
        import module_to_trace
    except ImportError:
        # If we don't handle the error, program will exit before
        # we get to inspect the events.
        pass

# Inspect the events after ending the trace
all_loaded = {e.path for e in events if isinstance(e, dlltracer.LoadEvent)}
all_failed = {e.path for e in events if isinstance(e, dlltracer.LoadFailedEvent)}

Raise audit events

To raise audit events for DLL loads, pass audit=True to Trace. The events raised are dlltracer.load and dlltracer.failed, and both only include the path as an argument.

import dlltracer
import sys

def hook(event, args):
    if event == "dlltracer.load":
        # args = (path,)
        print("Loaded", args[0])
    elif event == "dlltracer.failed":
        # args = (path,)
        print("Failed", args[0])

sys.add_audit_hook(hook)

with dlltracer.Trace(audit=True):
    import module_to_trace

Additional events

Note: This is mainly intended for development of dlltracer.

Because event formats may change, and additional events may be of interest but are not yet handled, passing the debug=True option to Trace enables all events to be collected, written, or audited. Regular events are suppressed.

import dlltracer
import sys

def hook(event, args):
    if event != "dlltracer.debug":
        return

    # args schema:
    #   provider is a UUID representing the event source
    #   opcode is an int representing the operation
    #   header is bytes taken directly from the event header
    #   data is bytes taken directly from the event data
    provider, opcode, header, data = args

sys.add_audit_hook(hook)

with dlltracer.Trace(debug=True, audit=True, collect=True, out=sys.stderr) as events:
    try:
        import module_to_trace
    except ImportError:
        pass

for e in events:
    assert isinstance(e, dlltracer.DebugEvent)
    # DebugEvent contains provider, opcode, header and data as for the audit event

Contribute

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

dlltracer-python's People

Contributors

microsoftopensource avatar zooba avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dlltracer-python's Issues

How to build this package?

I just discovered dlltracer and tried it on a PyCurl example.
It works really nice. Although I didn't like the:
LoadLibrary \Device\HarddiskVolume3\Windows\System32\kernel.appcore.dll syntax.
IMHO a LoadLibrary c:\Windows\System32\kernel.appcore.dll would be nicer. Possible?

To the issue; the README.md doesn't mention how to build it.
So I looked in .github/workflows/test.yml for clues and tried various ways that all failed. (in the git clone directory):

  • pyproject-build.exe --wheel . failed with:
 * Creating virtualenv isolated environment...
 * Installing packages in isolated environment... (Cython, pymsbuild>=1.0.0b6)
 * Getting build dependencies for wheel...
 * Building wheel...
   C:\temp\build-env-qfqufgzl\Scripts\python.exe: No module named cython.__main__; 
  'cython'  is a  package and cannot be directly executed
   C:\temp\build-env-qfqufgzl\lib\site-packages\pymsbuild\targets\cython.targets(54,5): error MSB3073: 
   The command ""C:\temp\build-env-qfqufgzl\Scripts\python.exe" -m cython  -o 
   F:\gv\Python310\Lib\site-packages\dlltracer\Git latest\build\temp\_native.cp310-win_amd64\src\dlltracer\_native.cpp 
  -f -v --cplus F:\gv\Python310\Lib\site-packages\dlltracer\Git latest\src\dlltracer\_native.pyx"
  exited with code 1. [F:\gv\Python310\Lib\site-packages\dlltracer\Git-latest\build\temp\_native.proj]

  ERROR Backend subprocess exited when trying to invoke build_wheel

Although I did install/upgraded to the latest versions of Cython + pymsbuild.

  • py.exe -3 -m pymsbuild -v also failed with the above No module named cython.__main__ .

  • Then trying with py.exe -3 -m pip install -vvv "git+https://github.com/microsoft/dlltracer-python.git" which also failed:

  Building wheels for collected packages: dlltracer
    Created temporary directory: C:\temp\pip-wheel-h_o60wpu
    Destination directory: C:\temp\pip-wheel-h_o60wpu
    Running command Building wheel for dlltracer (pyproject.toml)

  f:\gv\VC_2022\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(541,5): warning MSB8029: The 
  Intermediate directory or Output directory cannot reside under the Temporary directory as it could lead to issues 
  with incremental build. [C:\temp\pip-req-build-w1ibs1je\build\temp\_native.proj]
    F:\gv\Python310\python.exe: No module named cython.__main__; 'cython' is a package and cannot be 
  directly executed
  C:\temp\pip-build-env-hq6p9g50\overlay\Lib\site-packages\pymsbuild\targets\cython.targets(54,5): error MSB3073: 
  The command ""F:\gv\Python310\python.exe" -m cython  -o C:\temp\pip-req-build-w1ibs1je\build\temp\_native.cp310-win_amd64\src\dlltracer\_native.cpp -f -v  --cplus
 C:\temp\pip-req-build-w1ibs1je\src\dlltracer\_native.pyx" exited with code 1. 
[C:\temp\pip-req-build-w1ibs1je\build\temp\_native.proj]  error: subprocess-exited-with-error

  × Building wheel for dlltracer (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: 
'F:\gv\Python310\python.exe' 'F:\gv\Python310\lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py' 
  build_wheel 'c:\temp\tmp0dgu2c_z'
  cwd: C:\temp\pip-req-build-w1ibs1je
  Building wheel for dlltracer (pyproject.toml) ... error
  ERROR: Failed building wheel for dlltracer
Failed to build dlltracer
ERROR: Could not build wheels for dlltracer, which is required to install pyproject.toml-based projects
Exception information:
Traceback (most recent call last):
  File "F:\gv\Python310\lib\site-packages\pip\_internal\cli\base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)
  File "F:\gv\Python310\lib\site-packages\pip\_internal\cli\req_command.py", line 245, in wrapper
    return func(self, options, args)
  File "F:\gv\Python310\lib\site-packages\pip\_internal\commands\install.py", line 429, in run
    raise InstallationError(
pip._internal.exceptions.InstallationError: Could not build wheels for dlltracer, which is required to install pyproject.toml-base
d projects
Remote version of pip: 24.0
Local version of pip:  24.0
Was pip installed by pip? True
Removed build tracker: 'C:\\temp\\pip-build-tracker-shx_rvs7'

How hard should it be. Arg! What is going on here? Some MSbuild issue or what?
So I ended up with this build.bat:

  md build
  cython.exe -o build/_native.cpp -3f --cplus src/dlltracer/_native.pyx

  cl -nologo -MD -W3 -Ot -Zi -wd4244 -wd4551 -Fo./build\_native.obj -I./src -I%PY3_HOME%/include build/_native.cpp
  link -nologo -out:build/_native.pyd -dll -incremental:no -debug build/_native.obj -libpath:%PY3_HOME%/libs python3.lib advapi32.lib

that works for me.

PS. I have no issue with MSbuild itself from the command-line or via vcpkg.exe.
And I have the latest Visual-Studio 17.0 Preview 1.0. I.e. cl ver. 14.40.33521.

Resolve device paths

From #7:

I didn't like the:
LoadLibrary \Device\HarddiskVolume3\Windows\System32\kernel.appcore.dll syntax.
IMHO a LoadLibrary c:\Windows\System32\kernel.appcore.dll would be nicer. Possible?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.