course-ds-base's Issues
Task 2.2 - Set up the project repository structure
Branch: step-1-organize-ml-project
Tasks:
- checkout to the branch: step-1-organize-ml-project
- create the project structure
- commit & push to remote
Check yourself:
- branch: step-1-organize-ml-project_SOLUTION
Task 3.2 - Step 2: Create a single configuration file
Branch: step-2-create-config-file
Tasks:
- create
params.yaml
file - move configs into
params.yaml
Task 2.1 - Clone the tutorial repo and learn basic Git commands
Branch: main
Task:
- Fork & clone repo
- create a branch: git_basics
- checkout, create a file, commit & push to remote
- Checkout diffs for main VS. git_basics
- Create a PR and merge to main (on remote)
Task 3.4 - Step 4: Build ML experiment pipeline
Branch: step-4-build-ml-pipeline
Tasks:
-
create
src/stages
directory -
create
.py
modules for each pipeline stage:- data_load.py
- data_split.py
- featurize.py
- train.py
- evaluate.py
-
run each stage
Requirements:
- use
params.yaml
to manage stages configuration
Task 3.3 - Step 3: Move reusable code to .py modules
Branch: step-3-reusable-code
Tasks:
- move reusable code to .py modules
Task 4.1 - How data versioning works?
Branch: step-6-data-and-model-version-control
Tasks:
- create a
file.txt
- add
file.txt
to version control with DVC - create a
datadir
and add it to version control with DVC
Install fails on Ubuntu/Python 3.12
Had problems installing the project with python 3.12 on Ubuntu.
pyyaml and pandas failed to build.
pandas seemed to have a permissions issue when it was trying to use clang, so that might be an issue with my setup, but pyyaml seems to be a more general issue:
Collecting pyyaml<6,>=5.4.1 (from -r requirements.txt (line 10))
Using cached PyYAML-5.4.1.tar.gz (175 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [54 lines of output]
running egg_info
writing lib3/PyYAML.egg-info/PKG-INFO
writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
writing top-level names to lib3/PyYAML.egg-info/top_level.txt
Traceback (most recent call last):
File "/home/toonarmycaptain/PycharmProjects/ML-DVC/DVC/course-ds-base/dvc-venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
main()
File "/home/toonarmycaptain/PycharmProjects/ML-DVC/DVC/course-ds-base/dvc-venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/toonarmycaptain/PycharmProjects/ML-DVC/DVC/course-ds-base/dvc-venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-nbcjjmi0/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 332, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=[])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-nbcjjmi0/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 302, in _get_build_requires
self.run_setup()
File "/tmp/pip-build-env-nbcjjmi0/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 318, in run_setup
exec(code, locals())
File "<string>", line 271, in <module>
File "/tmp/pip-build-env-nbcjjmi0/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 111, in setup
return distutils.core.setup(**attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-nbcjjmi0/overlay/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 184, in setup
return run_commands(dist)
^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-nbcjjmi0/overlay/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
dist.run_commands()
File "/tmp/pip-build-env-nbcjjmi0/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 964, in run_commands
self.run_command(cmd)
File "/tmp/pip-build-env-nbcjjmi0/overlay/lib/python3.12/site-packages/setuptools/dist.py", line 948, in run_command
super().run_command(command)
File "/tmp/pip-build-env-nbcjjmi0/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 983, in run_command
cmd_obj.run()
File "/tmp/pip-build-env-nbcjjmi0/overlay/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 311, in run
self.find_sources()
File "/tmp/pip-build-env-nbcjjmi0/overlay/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 319, in find_sources
mm.run()
File "/tmp/pip-build-env-nbcjjmi0/overlay/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 540, in run
self.add_defaults()
File "/tmp/pip-build-env-nbcjjmi0/overlay/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 578, in add_defaults
sdist.add_defaults(self)
File "/tmp/pip-build-env-nbcjjmi0/overlay/lib/python3.12/site-packages/setuptools/command/sdist.py", line 105, in add_defaults
super().add_defaults()
File "/tmp/pip-build-env-nbcjjmi0/overlay/lib/python3.12/site-packages/setuptools/_distutils/command/sdist.py", line 250, in add_defaults
self._add_defaults_ext()
File "/tmp/pip-build-env-nbcjjmi0/overlay/lib/python3.12/site-packages/setuptools/_distutils/command/sdist.py", line 335, in _add_defaults_ext
self.filelist.extend(build_ext.get_source_files())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<string>", line 201, in get_source_files
File "/tmp/pip-build-env-nbcjjmi0/overlay/lib/python3.12/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
raise AttributeError(attr)
AttributeError: cython_sources
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Either way, the project requirements seem to install just fine by commenting out the version requirements:
(dvc-venv) toonarmycaptain$ python --version
Python 3.12.3
(dvc-venv) toonarmycaptain$ cat requirements.txt
dvc>=2.8.3,<3
joblib>=1.0.1,<2
jupyter>=1.0.0,<2
jupyter_contrib_nbextensions>=0.5.1,<1
matplotlib>=3.4.3,<4
numpy>=1.21.2,<2
#pandas>=1.3.2,<2
pandas
pytest>=6.2.4,<7
python-box>=5.4.1,<6
#pyyaml>=5.4.1,<6
pyyaml
scikit-learn>=0.24.2,<2
scipy>=1.7.1,<2
tqdm>=4.62.2,<5
Requirements could not deploy on Python3.10
@Team,
Trying to deploy requirements on Python3.10 version. Please let us know, is this specific specific python version have to deploy and test myself this code.
(dvc-venv) PS C:\Jan5-2021\DVC\course-ds-base> pip install -r .\requirements.txt
Collecting dvc==2.8.3
Downloading dvc-2.8.3-py3-none-any.whl (399 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 399.3/399.3 kB 2.3 MB/s eta 0:00:00
Collecting joblib==1.0.1
Downloading joblib-1.0.1-py3-none-any.whl (303 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 303.1/303.1 kB 6.2 MB/s eta 0:00:00
Collecting jupyter==1.0.0
Using cached jupyter-1.0.0-py2.py3-none-any.whl (2.7 kB)
Collecting jupyter_contrib_nbextensions==0.5.1
Downloading jupyter_contrib_nbextensions-0.5.1-py2.py3-none-any.whl (20.9 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.9/20.9 MB 1.3 MB/s eta 0:00:00
Collecting matplotlib==3.4.3
Downloading matplotlib-3.4.3.tar.gz (37.9 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 37.9/37.9 MB 1.3 MB/s eta 0:00:00
Preparing metadata (setup.py) ... done
Collecting numpy==1.21.2
Downloading numpy-1.21.2.zip (10.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.3/10.3 MB 1.6 MB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting pandas==1.3.2
Downloading pandas-1.3.2.tar.gz (4.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.7/4.7 MB 4.6 MB/s eta 0:00:00
Installing build dependencies ... error
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> [297 lines of output]
Ignoring numpy: markers 'python_version == "3.7" and (platform_machine != "arm64" or platform_system != "Darwin") and platform_machine != "aarch64
"' don't match your environment
Ignoring numpy: markers 'python_version == "3.8" and (platform_machine != "arm64" or platform_system != "Darwin") and platform_machine != "aarch64
"' don't match your environment
Ignoring numpy: markers 'python_version == "3.7" and platform_machine == "aarch64"' don't match your environment
Ignoring numpy: markers 'python_version == "3.8" and platform_machine == "aarch64"' don't match your environment
Ignoring numpy: markers 'python_version == "3.8" and platform_machine == "arm64" and platform_system == "Darwin"' don't match your environment
Ignoring numpy: markers 'python_version == "3.9" and platform_machine == "arm64" and platform_system == "Darwin"' don't match your environment
Collecting setuptools>=51.0.0
Using cached setuptools-65.5.0-py3-none-any.whl (1.2 MB)
Collecting wheel
Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Collecting Cython<3,>=0.29.21
Using cached Cython-0.29.32-py2.py3-none-any.whl (986 kB)
Collecting numpy==1.19.3
Downloading numpy-1.19.3.zip (7.3 MB)
---------------------------------------- 7.3/7.3 MB 485.5 kB/s eta 0:00:00
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'error'
error: subprocess-exited-with-error
Preparing metadata (pyproject.toml) did not run successfully.
exit code: 1
[259 lines of output]
setup.py:67: RuntimeWarning: NumPy 1.19.3 may not yet support Python 3.10.
warnings.warn(
Running from numpy source directory.
setup.py:480: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates
run_build = parse_setuppy_commands()
Processing numpy/random\_bounded_integers.pxd.in
Processing numpy/random\bit_generator.pyx
Processing numpy/random\mtrand.pyx
Processing numpy/random\_bounded_integers.pyx.in
Processing numpy/random\_common.pyx
Processing numpy/random\_generator.pyx
Processing numpy/random\_mt19937.pyx
Processing numpy/random\_pcg64.pyx
Processing numpy/random\_philox.pyx
Processing numpy/random\_sfc64.pyx
Cythonizing sources
blas_opt_info:
blas_mkl_info:
No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils
customize MSVCCompiler
libraries mkl_rt not found in ['C:\\Jan5-2021\\DVC\\course-ds-base\\dvc-venv\\lib', 'C:\\']
NOT AVAILABLE
blis_info:
libraries blis not found in ['C:\\Jan5-2021\\DVC\\course-ds-base\\dvc-venv\\lib', 'C:\\']
NOT AVAILABLE
openblas_info:
libraries openblas not found in ['C:\\Jan5-2021\\DVC\\course-ds-base\\dvc-venv\\lib', 'C:\\']
get_default_fcompiler: matching types: '['gnu', 'intelv', 'absoft', 'compaqv', 'intelev', 'gnu95', 'g95', 'intelvem', 'intelem', 'flang']'
customize GnuFCompiler
Could not locate executable g77
Could not locate executable f77
customize IntelVisualFCompiler
Could not locate executable ifort
Could not locate executable ifl
customize AbsoftFCompiler
Could not locate executable f90
customize CompaqVisualFCompiler
Could not locate executable DF
customize IntelItaniumVisualFCompiler
Could not locate executable efl
customize Gnu95FCompiler
Could not locate executable gfortran
Could not locate executable f95
customize G95FCompiler
Could not locate executable g95
customize IntelEM64VisualFCompiler
customize IntelEM64TFCompiler
Could not locate executable efort
Could not locate executable efc
customize PGroupFlangCompiler
Could not locate executable flang
don't know how to compile Fortran code on platform 'nt'
NOT AVAILABLE
atlas_3_10_blas_threads_info:
Setting PTATLAS=ATLAS
libraries tatlas not found in ['C:\\Jan5-2021\\DVC\\course-ds-base\\dvc-venv\\lib', 'C:\\']
NOT AVAILABLE
atlas_3_10_blas_info:
libraries satlas not found in ['C:\\Jan5-2021\\DVC\\course-ds-base\\dvc-venv\\lib', 'C:\\']
NOT AVAILABLE
atlas_blas_threads_info:
Setting PTATLAS=ATLAS
libraries ptf77blas,ptcblas,atlas not found in ['C:\\Jan5-2021\\DVC\\course-ds-base\\dvc-venv\\lib', 'C:\\']
NOT AVAILABLE
atlas_blas_info:
libraries f77blas,cblas,atlas not found in ['C:\\Jan5-2021\\DVC\\course-ds-base\\dvc-venv\\lib', 'C:\\']
NOT AVAILABLE
accelerate_info:
NOT AVAILABLE
C:\Users\seerapu\AppData\Local\Temp\pip-install-jsn8okah\numpy_c87dffe387db45458072f04c8df762c4\numpy\distutils\system_info.py:1914: UserWarning
:
Optimized (vendor) Blas libraries are not found.
Falls back to netlib Blas library which has worse performance.
A better performance should be easily gained by switching
Blas library.
if self._calc_info(blas):
blas_info:
libraries blas not found in ['C:\Jan5-2021\DVC\course-ds-base\dvc-venv\lib', 'C:\']
NOT AVAILABLE
C:\Users\seerapu\AppData\Local\Temp\pip-install-jsn8okah\numpy_c87dffe387db45458072f04c8df762c4\numpy\distutils\system_info.py:1914: UserWarning
:
Blas (http://www.netlib.org/blas/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [blas]) or by setting
the BLAS environment variable.
if self._calc_info(blas):
blas_src_info:
NOT AVAILABLE
C:\Users\seerapu\AppData\Local\Temp\pip-install-jsn8okah\numpy_c87dffe387db45458072f04c8df762c4\numpy\distutils\system_info.py:1914: UserWarning
:
Blas (http://www.netlib.org/blas/) sources not found.
Directories to search for the sources can be specified in the
numpy/distutils/site.cfg file (section [blas_src]) or by setting
the BLAS_SRC environment variable.
if self._calc_info(blas):
NOT AVAILABLE
non-existing path in 'numpy\\distutils': 'site.cfg'
lapack_opt_info:
lapack_mkl_info:
libraries mkl_rt not found in ['C:\\Jan5-2021\\DVC\\course-ds-base\\dvc-venv\\lib', 'C:\\']
NOT AVAILABLE
openblas_lapack_info:
libraries openblas not found in ['C:\\Jan5-2021\\DVC\\course-ds-base\\dvc-venv\\lib', 'C:\\']
NOT AVAILABLE
openblas_clapack_info:
libraries openblas,lapack not found in ['C:\\Jan5-2021\\DVC\\course-ds-base\\dvc-venv\\lib', 'C:\\']
NOT AVAILABLE
flame_info:
libraries flame not found in ['C:\\Jan5-2021\\DVC\\course-ds-base\\dvc-venv\\lib', 'C:\\']
NOT AVAILABLE
atlas_3_10_threads_info:
Setting PTATLAS=ATLAS
libraries lapack_atlas not found in C:\Jan5-2021\DVC\course-ds-base\dvc-venv\lib
libraries tatlas,tatlas not found in C:\Jan5-2021\DVC\course-ds-base\dvc-venv\lib
libraries lapack_atlas not found in C:\
libraries tatlas,tatlas not found in C:\
<class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
NOT AVAILABLE
atlas_3_10_info:
libraries lapack_atlas not found in C:\Jan5-2021\DVC\course-ds-base\dvc-venv\lib
libraries satlas,satlas not found in C:\Jan5-2021\DVC\course-ds-base\dvc-venv\lib
libraries lapack_atlas not found in C:\
libraries satlas,satlas not found in C:\
<class 'numpy.distutils.system_info.atlas_3_10_info'>
NOT AVAILABLE
atlas_threads_info:
Setting PTATLAS=ATLAS
libraries lapack_atlas not found in C:\Jan5-2021\DVC\course-ds-base\dvc-venv\lib
libraries ptf77blas,ptcblas,atlas not found in C:\Jan5-2021\DVC\course-ds-base\dvc-venv\lib
libraries lapack_atlas not found in C:\
libraries ptf77blas,ptcblas,atlas not found in C:\
<class 'numpy.distutils.system_info.atlas_threads_info'>
NOT AVAILABLE
atlas_info:
libraries lapack_atlas not found in C:\Jan5-2021\DVC\course-ds-base\dvc-venv\lib
libraries f77blas,cblas,atlas not found in C:\Jan5-2021\DVC\course-ds-base\dvc-venv\lib
libraries lapack_atlas not found in C:\
libraries f77blas,cblas,atlas not found in C:\
<class 'numpy.distutils.system_info.atlas_info'>
NOT AVAILABLE
lapack_info:
libraries lapack not found in ['C:\\Jan5-2021\\DVC\\course-ds-base\\dvc-venv\\lib', 'C:\\']
NOT AVAILABLE
C:\Users\seerapu\AppData\Local\Temp\pip-install-jsn8okah\numpy_c87dffe387db45458072f04c8df762c4\numpy\distutils\system_info.py:1748: UserWarning
:
Lapack (http://www.netlib.org/lapack/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [lapack]) or by setting
the LAPACK environment variable.
return getattr(self, 'calc_info{}'.format(name))()
lapack_src_info:
NOT AVAILABLE
C:\Users\seerapu\AppData\Local\Temp\pip-install-jsn8okah\numpy_c87dffe387db45458072f04c8df762c4\numpy\distutils\system_info.py:1748: UserWarning
:
Lapack (http://www.netlib.org/lapack/) sources not found.
Directories to search for the sources can be specified in the
numpy/distutils/site.cfg file (section [lapack_src]) or by setting
the LAPACK_SRC environment variable.
return getattr(self, 'calc_info{}'.format(name))()
NOT AVAILABLE
numpy_linalg_lapack_lite:
FOUND:
language = c
define_macros = [('HAVE_BLAS_ILP64', None), ('BLAS_SYMBOL_SUFFIX', '64_')]
C:\Users\seerapu\AppData\Local\Temp\pip-build-env-r0sar12d\overlay\Lib\site-packages\setuptools\_distutils\dist.py:275: UserWarning: Unknown dis
tribution option: 'define_macros'
warnings.warn(msg)
running dist_info
running build_src
build_src
building py_modules sources
creating build
creating build\src.win-amd64-3.10
creating build\src.win-amd64-3.10\numpy
creating build\src.win-amd64-3.10\numpy\distutils
building library "npymath" sources
Traceback (most recent call last):
File "C:\Jan5-2021\DVC\course-ds-base\dvc-venv\Lib\site-packages\pip_vendor\pep517\in_process_in_process.py", line 351, in
main()
File "C:\Jan5-2021\DVC\course-ds-base\dvc-venv\Lib\site-packages\pip_vendor\pep517\in_process_in_process.py", line 333, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "C:\Jan5-2021\DVC\course-ds-base\dvc-venv\Lib\site-packages\pip_vendor\pep517\in_process_in_process.py", line 152, in prepare_metadata_
for_build_wheel
return hook(metadata_directory, config_settings)
File "C:\Users\seerapu\AppData\Local\Temp\pip-build-env-r0sar12d\overlay\Lib\site-packages\setuptools\build_meta.py", line 157, in prepare_met
adata_for_build_wheel
self.run_setup()
File "C:\Users\seerapu\AppData\Local\Temp\pip-build-env-r0sar12d\overlay\Lib\site-packages\setuptools\build_meta.py", line 248, in run_setup
super(BuildMetaLegacyBackend,
File "C:\Users\seerapu\AppData\Local\Temp\pip-build-env-r0sar12d\overlay\Lib\site-packages\setuptools\build_meta.py", line 142, in run_setup
exec(compile(code, file, 'exec'), locals())
File "setup.py", line 508, in
setup_package()
File "setup.py", line 500, in setup_package
setup(**metadata)
File "C:\Users\seerapu\AppData\Local\Temp\pip-install-jsn8okah\numpy_c87dffe387db45458072f04c8df762c4\numpy\distutils\core.py", line 169, in s
etup
return old_setup(**new_attr)
File "C:\Users\seerapu\AppData\Local\Temp\pip-build-env-r0sar12d\overlay\Lib\site-packages\setuptools_init.py", line 165, in setup
return distutils.core.setup(**attrs)
File "C:\Users\seerapu\AppData\Local\Temp\pip-build-env-r0sar12d\overlay\Lib\site-packages\setuptools_distutils\core.py", line 148, in setup
dist.run_commands()
File "C:\Users\seerapu\AppData\Local\Temp\pip-build-env-r0sar12d\overlay\Lib\site-packages\setuptools_distutils\dist.py", line 967, in run_co
mmands
self.run_command(cmd)
File "C:\Users\seerapu\AppData\Local\Temp\pip-build-env-r0sar12d\overlay\Lib\site-packages\setuptools_distutils\dist.py", line 986, in run_co
mmand
cmd_obj.run()
File "C:\Users\seerapu\AppData\Local\Temp\pip-build-env-r0sar12d\overlay\Lib\site-packages\setuptools\command\dist_info.py", line 31, in run
egg_info.run()
File "C:\Users\seerapu\AppData\Local\Temp\pip-install-jsn8okah\numpy_c87dffe387db45458072f04c8df762c4\numpy\distutils\command\egg_info.py", li
ne 24, in run
self.run_command("build_src")
File "C:\Users\seerapu\AppData\Local\Temp\pip-build-env-r0sar12d\overlay\Lib\site-packages\setuptools_distutils\cmd.py", line 313, in run_com
mand
self.distribution.run_command(command)
File "C:\Users\seerapu\AppData\Local\Temp\pip-build-env-r0sar12d\overlay\Lib\site-packages\setuptools_distutils\dist.py", line 986, in run_co
mmand
cmd_obj.run()
File "C:\Users\seerapu\AppData\Local\Temp\pip-install-jsn8okah\numpy_c87dffe387db45458072f04c8df762c4\numpy\distutils\command\build_src.py", l
ine 144, in run
self.build_sources()
File "C:\Users\seerapu\AppData\Local\Temp\pip-install-jsn8okah\numpy_c87dffe387db45458072f04c8df762c4\numpy\distutils\command\build_src.py", l
ine 155, in build_sources
self.build_library_sources(libname_info)
File "C:\Users\seerapu\AppData\Local\Temp\pip-install-jsn8okah\numpy_c87dffe387db45458072f04c8df762c4\numpy\distutils\command\build_src.py", l
ine 288, in build_library_sources
sources = self.generate_sources(sources, (lib_name, build_info))
File "C:\Users\seerapu\AppData\Local\Temp\pip-install-jsn8okah\numpy_c87dffe387db45458072f04c8df762c4\numpy\distutils\command\build_src.py", l
ine 378, in generate_sources
source = func(extension, build_dir)
File "numpy\core\setup.py", line 658, in get_mathlib_info
st = config_cmd.try_link('int main(void) { return 0;}')
File "C:\Users\seerapu\AppData\Local\Temp\pip-build-env-r0sar12d\overlay\Lib\site-packages\setuptools_distutils\command\config.py", line 243,
in try_link
self._link(body, headers, include_dirs,
File "C:\Users\seerapu\AppData\Local\Temp\pip-install-jsn8okah\numpy_c87dffe387db45458072f04c8df762c4\numpy\distutils\command\config.py", line
162, in _link
return self._wrap_method(old_config._link, lang,
File "C:\Users\seerapu\AppData\Local\Temp\pip-install-jsn8okah\numpy_c87dffe387db45458072f04c8df762c4\numpy\distutils\command\config.py", line
96, in _wrap_method
ret = mth(((self,)+args))
File "C:\Users\seerapu\AppData\Local\Temp\pip-build-env-r0sar12d\overlay\Lib\site-packages\setuptools_distutils\command\config.py", line 137,
in _link
(src, obj) = self._compile(body, headers, include_dirs, lang)
File "C:\Users\seerapu\AppData\Local\Temp\pip-install-jsn8okah\numpy_c87dffe387db45458072f04c8df762c4\numpy\distutils\command\config.py", line
105, in _compile
src, obj = self._wrap_method(old_config._compile, lang,
File "C:\Users\seerapu\AppData\Local\Temp\pip-install-jsn8okah\numpy_c87dffe387db45458072f04c8df762c4\numpy\distutils\command\config.py", line
96, in _wrap_method
ret = mth(*((self,)+args))
File "C:\Users\seerapu\AppData\Local\Temp\pip-build-env-r0sar12d\overlay\Lib\site-packages\setuptools_distutils\command\config.py", line 132,
in _compile
self.compiler.compile([src], include_dirs=include_dirs)
File "C:\Users\seerapu\AppData\Local\Temp\pip-build-env-r0sar12d\overlay\Lib\site-packages\setuptools_distutils_msvccompiler.py", line 401,
in compile
self.spawn(args)
File "C:\Users\seerapu\AppData\Local\Temp\pip-build-env-r0sar12d\overlay\Lib\site-packages\setuptools_distutils_msvccompiler.py", line 505,
in spawn
return super().spawn(cmd, env=env)
File "C:\Users\seerapu\AppData\Local\Temp\pip-install-jsn8okah\numpy_c87dffe387db45458072f04c8df762c4\numpy\distutils\ccompiler.py", line 90,
in
m = lambda self, *args, **kw: func(self, *args, **kw)
TypeError: CCompiler_spawn() got an unexpected keyword argument 'env'
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
Encountered error while generating package metadata.
See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Task 3.6 - Reproduce end-to-end ML pipelines
Branch: step-5-automate-ml-pipeline
Tasks:
- make sure that you've completed the Task 3.5
- run
dvc repro
(assume we do this for 1st time) - run
dvc repro
(no changes, DVC skips all stages) - run
dvc repro -f
(force running ML pipeline) - run
dvc repro train
(run onlytrain
stage fromdvc.yaml
) - update
train
stage configs and rundvc repro
- update
src/stage/train.py
code and rundvc repro
- remove
reports/metrics.json
and rundvc repro
Task 5.1 - Metrics Tracking with DVC
Branch: step-7-metrics-and-experiments
Tasks:
- save 'reports/metrics.json` file
- specify
metrics
indvc.yaml
-
dvc metrics show
-
dvc metrics diff
Task 4.3 - Tracking changes and switching between versions
Branch: step-6-data-and-model-version-control
(continue from #11)
Tasks:
- add remote storage
- store
file.txt
to remote storage - delete
file.txt
in workspace - restore
file.txt
from DVC cache withdvc checkout file.txt
- clean cache with
dvc gc
- try to restore
file.txt
from DVC cache withdvc checkout file.txt
(failure is expected) - restore
file.txt
from DVC Remote Storagedvc pull
(success is expected) - update 'file.txt' (version 2) and push it to remote storage
- switch btw Git branches and 'file.txt' versions
Task 3.5 - Step 5: Automate pipelines with DVC
Branch: step-5-automate-ml-pipeline
Tasks:
- install DVC (in a virtual environment)
- automate ML pipeline with DVC
- setup dependencies and outputs
- setup parameters
Requirements:
- for each stage, explicitly define dependencies (code and data)
Task 2.3 - Set up Python Virtual Environment
Branch: step-1-organize-ml-project
Tasks:
- create venv
- install python dependencies
- activate the environment and run Jupyter Notebook
Task 3.1 - Step 1: Organize repo & build a prototype
Branch: step-1-organize-ml-project
Tasks:
- Build a prototype with Jupyter Notebook
Task 4.4 - Data Access
Branch: step-6-data-and-model-version-control
Tasks:
- show example of
dvc list
anddvc get
commands
Task 2.4 - Discuss documentation & task tracking
Branch: step-1-organize-ml-project
Tasks:
- Show README, typing & docstring
- Use GitHub issues for task tracking
Task 4.2 - Version and share data and models. Remote storage
Branch: step-6-data-and-model-version-control
(continue from #11)
Tasks:
- add remote storage
- push data and models controlled by DVC to remote storage with
dvc push
- delete files in workspace
- restore siles from DVC Remote Storage
dvc pull
Task 5.2 - Plots and graphics with DVC
Branch: step-7-metrics-and-experiments
Tasks:
- save 'reports/classes.csv` file
- get confusion matrix plot with
dvc plots show reports/classes.csv --template confusion --template confusion
-
dvc plots show
-
dvc plots diff
Task 5.3 - Metrics Tracking and Plots with Studio
Branch: step-9-studio
Tasks:
- connect a GitHub repo
- walk through comparing metrics, params, plots, and running new experiments
Task 6.1 - Experimenting Workflow
Branch: step-9-experimenting-workflow
Tasks:
Experiment 1
- run a new experiment with updated parameter:
dvc exp run -S train.cv=2
- show metrics with CLI:
dvc exp show
- apply to the current branch / commit & push to remote /
dvc exp apply <exp>
git add . & git commit -m "Experiment 1: train.cv=2"
Experiment 2
- run a new experiment
dvc exp run -S train.estimator_name=svm
- create a branch / commit / push to remote / check updates in Studio
dvc exp branch <exp> experiment-2
git add . & git commit -m "Experiment 2: SVM"
git push origin experiment-2
- merge to step-9-experimenting / check updates in Studio
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.