ansys / pytwin Goto Github PK

📝 Description of the feature

PyTwin does not support import by multiple python processes.

This is due to the way the logging and working directory are setup by pytwin.settings when importing pytwin in a given python processes.

The feature will be to let user import pytwin in multiple python processes running at the same time and call any public method from the pytwin.settings module in such context.

💡 Steps for implementing the feature

One possible quick solution would be to add the python process id in the working directory path (see pytwin.settings.PYTWIN_SETTINGS.working_dir and how pytwin.evaluate.model.model_dir is working with it).

One difficulty is to handle the working directory migration when pytwin.settings.modify_pytwin_working_dir is called.

This should be refined based on the contexts where this issue has been encountered:

- SAF context

To be described (@ansJBoly could you please do so? Thanks!)

- Digital Twin context

To be described (@EDCarman could you please do so? Thanks!)

🔗 Useful links and references

No response

🔍 Before submitting the issue

• I have searched among the existing issues
• I am using a Python virtual environment

🐞 Description of the bug

The named_selection argument is the first argument for the function and is not optional. This is inconsistent with other similar functions e.g. generate_snapshot().

Additionally, the docstring lists named_selection as being the third argument and optional.

📝 Steps to reproduce

from pytwin import TwinModel
model1 = TwinModel('model.twin')
model1.initialize_evaluation()
romname = model1.tbrom_names[0]
nslist = model1.get_named_selections(romname)
# Generate a snapshot
fieldresults = model1.generate_snapshot(romname, False, nslist[0])
# Try to generate points with same arguments
points= model1.generate_snapshot(romname, False, nslist[0])

💻 Which operating system are you using?

Windows

📀 Which ANSYS version are you using?

No response

🐍 Which Python version are you using?

3.9

📦 Installed packages

colorama==0.4.6
contourpy==1.1.0
coverage==7.2.7
cycler==0.11.0
exceptiongroup==1.1.2
fonttools==4.41.0
importlib-metadata==6.8.0
importlib-resources==6.0.0
iniconfig==2.0.0
kiwisolver==1.4.4
matplotlib==3.7.2
numpy==1.25.1
packaging==23.1
pandas==2.0.3
Pillow==10.0.0
pluggy==1.2.0
pyparsing==3.0.9
pytest==7.4.0
pytest-cov==4.1.0
python-dateutil==2.8.2
-e git+https://github.com/pyansys/pytwin@ec3594c2f4f343d36dfa12bb35e6e6e78fffc638#egg=pytwin
pytz==2023.3
pywin32==306
six==1.16.0
tomli==2.0.1
tzdata==2023.3
zipp==3.16.1

🔍 Before submitting the issue

• I have searched among the existing issues
• I am using a Python virtual environment

🐞 Description of the bug

When instantiating a TwinModel in a loop, the allocated memory keep increasing.

📝 Steps to reproduce

import os
import tracemalloc
from pytwin import TwinModel

tracemalloc.start()
snapshot = tracemalloc.take_snapshot()

twin_file = os.path.join("path_to_a_twin_model_with_tbrom.twin")
for i in range(1, 100):
twin_model = TwinModel(model_filepath=twin_file)
snapshot2 = tracemalloc.take_snapshot()
top_stats = snapshot2.compare_to(snapshot, 'lineno')
stat_tbrom_read_basis = top_stats[0]
print(stat_tbrom_read_basis)

💻 Which operating system are you using?

Windows

📀 Which ANSYS version are you using?

NA

🐍 Which Python version are you using?

3.9

📦 Installed packages

certifi==2023.7.22
charset-normalizer==3.3.2
colorama==0.4.6
coverage==7.3.2
cycler==0.11.0
exceptiongroup==1.1.3    
fonttools==4.38.0        
idna==3.4
imageio==2.31.2
iniconfig==2.0.0
kiwisolver==1.4.5        
matplotlib==3.5.3        
numpy==1.21.1
packaging==23.2
pandas==1.3.5
Pillow==9.5.0
platformdirs==4.0.0      
pluggy==1.3.0
pooch==1.8.0
pyparsing==3.1.1
pytest==7.4.3
pytest-cov==4.1.0
python-dateutil==2.8.2
-e git+https://github.com/ansys/pytwin@16cf78a84ed27a2881d0c7781a8afe66750a76e9#egg=pytwin
pytz==2023.3.post1
pyvista==0.38.6
pywin32==306
requests==2.31.0
scooby==0.7.3
six==1.16.0
tomli==2.0.1
tqdm==4.66.1
urllib3==2.0.7
vtk==9.3.0

• Since pytwin is making use of actions, add the version switcher conf in docs.
• Add tox.ini file
• Add dependabot and labels
• rearrange the actions

📝 Description of the feature

Add option to read input snapshots from memory, rather than disk

For much the same reasons why we may prefer output snapshots in memory, rather than disk, we should consider providing input snapshots from memory. This will reduce disk I/O in cases where we may connect multiple models, all with vector inputs and outputs.

💡 Steps for implementing the feature

One potential implementation would be to modify the TbRom._reduce_field_input() method and replace the snapshot_filepath argument with a general snapshot argument that could be a string or Path, to read from disk, or an array or list of vec values.
Optionally add a on_disk: bool to explicitly indicate to treat the input as a vector or Path.

If on_disk, read the binary file as before. If not, skip reading and use the vector directly (convert list to np.ndarray?)

🔗 Useful links and references

No response

📝 Description of the feature

Currently, when a twin model is instantiated, it takes the global Pytwin log level. The twin runtime is instantiated with log_level=self._get_runtime_log_level().
It would be desirable to have the option to assign log levels on a per-instance basis. This is already possible at the core runtime level.

The current behaviour is problematic in the following case:
Certain twin models can produce extremely verbose logs at levels of INFO or higher, for example, containing detailed solver convergence logs for every time step. These messages quickly overwhelm any log messages from other parts of the code.

One workaround is to use a WARNING or lower log level, but then we lose INFO and DEBUG messages from other parts of the code.

Alternatively, we can use modify_pytwin_logging() to temporarily change the global log level to the desired level before instantiating the twin model e.g.

current_level = pytwin.settings.get_pytwin_log_level()
pytwin.modify_pytwin_logging(new_option=None, new_level='INFO')
twin_model = pytwin.TwinModel(model_path)
pytwin.modify_pytwin_logging(new_option=None, new_level=current_level)

💡 Steps for implementing the feature

We could add an optional log_level argument to the TwinModel __init__() method. This can then be propagated down to _instantiate_twin_model() and use when instantiating TwinRuntime.

The default value for the argument could be set to retain the current behaviour.

Something along the lines of:

class TwinModel(Model):
    ...
    def __init__(self, model_filepath: str, log_level=None):
        ...
        self._instantiate_twin_model(log_level)

    ...
    def _instantiate_twin_model(self, log_level=None):
        ...
        try:
            if log_level is None:
                log_level = self._get_runtime_log_level()
            ...
            # Instantiate twin runtime
            self._twin_runtime = TwinRuntime(
                model_path=self._model_filepath,
                load_model=True,
                log_path=self.model_log,
                log_level=log_level,
            )

🔗 Useful links and references

No response

Hi all!

As per @MaxJPRey request, I tried to add pytwin to the pyansys metapackage repository, so that it is distributed as part of our bundle package.

However, we encountered some issues in the process. It seems that pytwin does not support Python 3.7. This is a requirement for the metapackage since we are ensuring that all Python versions that are currently maintained keep being supported. This means Python 3.7 to 3.10. In fact, we are in process of supporting Python 3.11 as well - but this may take a while.

Would it be possible to make pytwin support Python 3.7? I can help you out implementing the needed changes. Thanks in advance!! 😃

🔍 Before submitting the issue

• I have searched among the existing issues
• I am using a Python virtual environment

🐞 Description of the bug

https://nvd.nist.gov/vuln/detail/CVE-2022-37434

📝 Steps to reproduce

pip install pytwin
Scan the resulting venv. The following vulnerability is found from use of madler-zlib 1.2.12
https://nvd.nist.gov/vuln/detail/CVE-2022-37434

💻 Which operating system are you using?

Windows

📀 Which ANSYS version are you using?

N/A

🐍 Which Python version are you using?

3.10

📦 Installed packages

numpy==1.26.1
pandas==2.1.1
python-dateutil==2.8.2
pytwin==0.5.0
pytz==2023.3.post1
pywin32==306
six==1.16.0
tzdata==2023.3

🔍 Before submitting the issue

• I have searched among the existing issues
• I am using a Python virtual environment

🐞 Description of the bug

Running the 3D Field Rom example, I am facing some errors related to the point cloud-based ROM Viewer embedded in the twin runtime.

When the model is initialised and the first output is updated, with the following code:

twin_model = TwinModel(twin_file)

twin_model.initialize_evaluation(inputs=rom_inputs, parameters=rom_parameters)

rom_name = twin_model.tbrom_names[0]
rom_directory = twin_model.get_rom_directory(rom_name)
snapshot = twin_model.get_snapshot_filepath(rom_name)
geometry = twin_model.get_geometry_filepath(rom_name)

temperature_file = snapshot_to_cfd(snapshot, geometry, "temperature", os.path.join(rom_directory, "cfd_file.ip"))

the following errors are logged:

dlopen1:libEGL.so.1: cannot open shared object file: No such file or directory
dlopen2:./RomViewerSharedLib.so: cannot open shared object file: No such file or directory

I can also find the following lines logged in the model log file:

[INFO] [ThermalTBROM_23R1SP1] [EXTME2][ThermalROM23R1_1][ThermalROM23R1_1][ThermalROM23R1_1] SetupExperiment [t = 0s] [Aug 16, 2023 02:50:04 PM]
[INFO] [ThermalTBROM_23R1SP1] [ME2FMU][ThermalROM23R1_1][ThermalROM23R1_1] fmi2EnterInitializationMode [t = 0s] [Aug 16, 2023 02:50:04 PM]
[INFO] [ThermalTBROM_23R1SP1] [ME2FMU][ThermalROM23R1_1][ThermalROM23R1_1] RomViewerSharedLib instance:00000000
 [t = 0s] [Aug 16, 2023 02:50:04 PM]
[WARNING] [ThermalTBROM_23R1SP1] [ME2FMU][ThermalROM23R1_1][ThermalROM23R1_1] Cannot Load RomViewer Library => unable to render figures... [t = 0s] [Aug 16, 2023 02:50:04 PM]

As a consequence, no image file is generated in the ROM directory and the following cell fails to display the image:

view_name = twin_model.get_available_view_names(rom_name)[0]
image_filepath = twin_model.get_image_filepath(rom_name, view_name)
plt.imshow(img.imread(image_filepath))
plt.show()

I suppose that some .so are missing and I was expecting them to be installed with the pytwin package, as this ROM viewer should be embedded in the runtime.

📝 Steps to reproduce

In a fresh environment, i.e. without any specific library added to the classpath, install pytwin and then run the 3D Field ROM example.

💻 Which operating system are you using?

Linux

📀 Which ANSYS version are you using?

0.3.0

🐍 Which Python version are you using?

3.10

📦 Installed packages

absl-py==1.0.0
accelerate==0.19.0
aiohttp==3.8.4
aiosignal==1.3.1
appdirs==1.4.4
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
astor==0.8.1
asttokens==2.2.1
astunparse==1.6.3
async-timeout==4.0.2
attrs==21.4.0
audioread==3.0.0
azure-core==1.27.1
azure-cosmos==4.3.1b1
azure-storage-blob==12.17.0b1
azure-storage-file-datalake==12.11.0
backcall==0.2.0
bcrypt==3.2.0
beautifulsoup4==4.11.1
black==22.6.0
bleach==4.1.0
blinker==1.4
blis==0.7.9
boto3==1.24.28
botocore==1.27.28
cachetools==4.2.4
catalogue==2.0.8
category-encoders==2.6.0
certifi==2022.9.14
cffi==1.15.1
chardet==4.0.0
charset-normalizer==2.0.4
click==8.0.4
cloudpickle==2.0.0
cmdstanpy==1.1.0
confection==0.0.4
configparser==5.2.0
convertdate==2.4.0
cryptography==37.0.1
cycler==0.11.0
cymem==2.0.7
Cython==0.29.32
dacite==1.8.1
databricks-automl-runtime==0.2.16
databricks-cli==0.17.7
databricks-feature-store==0.13.5
databricks-sdk==0.1.6
dataclasses-json==0.5.8
datasets==2.12.0
dbl-tempo==0.1.23
dbus-python==1.2.18
debugpy==1.5.1
decorator==5.1.1
defusedxml==0.7.1
dill==0.3.4
diskcache==5.6.1
distlib==0.3.6
distro==1.7.0
distro-info===1.1build1
docstring-to-markdown==0.12
entrypoints==0.4
ephem==4.1.4
evaluate==0.4.0
executing==1.2.0
facets-overview==1.0.3
fastjsonschema==2.17.1
fasttext==0.9.2
filelock==3.6.0
Flask @ https://databricks-build-artifacts-manual-staging.s3.amazonaws.com/flask/Flask-1.1.2%2Bdb1-py2.py3-none-any.whl?AWSAccessKeyId=AKIAX7HWM34HCSVHYQ7M&Expires=2001354391&Signature=bztIumr2jXFbisF0QicZvqbvT9s%3D
flatbuffers==23.5.26
fonttools==4.25.0
frozenlist==1.3.3
fsspec==2022.7.1
future==0.18.2
gast==0.4.0
gitdb==4.0.10
GitPython==3.1.27
google-api-core==2.8.2
google-auth==1.33.0
google-auth-oauthlib==0.4.6
google-cloud-core==2.3.2
google-cloud-storage==2.9.0
google-crc32c==1.5.0
google-pasta==0.2.0
google-resumable-media==2.5.0
googleapis-common-protos==1.56.4
greenlet==1.1.1
grpcio==1.48.1
grpcio-status==1.48.1
gunicorn==20.1.0
gviz-api==1.10.0
h5py==3.7.0
holidays==0.25
horovod==0.28.0
htmlmin==0.1.12
httplib2==0.20.2
huggingface-hub==0.15.1
idna==3.3
ImageHash==4.3.1
imbalanced-learn==0.8.1
importlib-metadata==4.11.3
importlib-resources==5.12.0
ipykernel==6.17.1
ipython==8.10.0
ipython-genutils==0.2.0
ipywidgets==7.7.2
isodate==0.6.1
itsdangerous==2.0.1
jedi==0.18.1
jeepney==0.7.1
Jinja2==2.11.3
jmespath==0.10.0
joblib==1.2.0
joblibspark==0.5.1
jsonschema==4.16.0
jupyter-client==7.3.4
jupyter_core==4.11.2
jupyterlab-pygments==0.1.2
jupyterlab-widgets==1.0.0
keras==2.11.0
keyring==23.5.0
kiwisolver==1.4.2
korean-lunar-calendar==0.3.1
langchain==0.0.181
langcodes==3.3.0
launchpadlib==1.10.16
lazr.restfulclient==0.14.4
lazr.uri==1.0.6
lazy_loader==0.2
libclang==15.0.6.1
librosa==0.10.0
lightgbm==3.3.5
llvmlite==0.38.0
LunarCalendar==0.0.9
Mako==1.2.0
Markdown==3.3.4
MarkupSafe==2.0.1
marshmallow==3.19.0
marshmallow-enum==1.5.1
matplotlib==3.5.2
matplotlib-inline==0.1.6
mccabe==0.7.0
mistune==0.8.4
mleap==0.20.0
mlflow-skinny==2.4.2
more-itertools==8.10.0
msgpack==1.0.5
multidict==6.0.4
multimethod==1.9.1
multiprocess==0.70.12.2
murmurhash==1.0.9
mypy-extensions==0.4.3
nbclient==0.5.13
nbconvert==6.4.4
nbformat==5.5.0
nest-asyncio==1.5.5
networkx==2.8.4
ninja==1.11.1
nltk==3.7
nodeenv==1.8.0
notebook==6.4.12
numba==0.55.1
numexpr==2.8.4
numpy==1.21.5
oauthlib==3.2.0
openai==0.27.7
openapi-schema-pydantic==1.2.4
opt-einsum==3.3.0
packaging==21.3
pandas==1.4.4
pandocfilters==1.5.0
paramiko==2.9.2
parso==0.8.3
pathspec==0.9.0
pathy==0.10.1
patsy==0.5.2
petastorm==0.12.1
pexpect==4.8.0
phik==0.12.3
pickleshare==0.7.5
Pillow==9.2.0
platformdirs==2.5.2
plotly==5.9.0
pluggy==1.0.0
pmdarima==2.0.3
pooch==1.7.0
preshed==3.0.8
prometheus-client==0.14.1
prompt-toolkit==3.0.36
prophet==1.1.3
protobuf==3.19.4
psutil==5.9.0
psycopg2==2.9.3
ptyprocess==0.7.0
pure-eval==0.2.2
pyarrow==8.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pybind11==2.10.4
pycparser==2.21
pydantic==1.10.6
pyflakes==3.0.1
Pygments==2.11.2
PyGObject==3.42.1
PyJWT==2.3.0
PyMeeus==0.5.12
PyNaCl==1.5.0
pyodbc==4.0.32
pyparsing==3.0.9
pyright==1.1.294
pyrsistent==0.18.0
pytesseract==0.3.10
python-apt==2.4.0+ubuntu1
python-dateutil==2.8.2
python-editor==1.0.4
python-lsp-jsonrpc==1.0.0
python-lsp-server==1.7.1
pytoolconfig==1.2.2
pytwin==0.3.0
pytz==2022.1
PyWavelets==1.3.0
PyYAML==6.0
pyzmq==23.2.0
regex==2022.7.9
requests==2.28.1
requests-oauthlib==1.3.1
responses==0.18.0
rope==1.7.0
rsa==4.9
s3transfer==0.6.0
scikit-learn==1.1.1
scipy==1.9.1
seaborn==0.11.2
SecretStorage==3.3.1
Send2Trash==1.8.0
sentence-transformers==2.2.2
sentencepiece==0.1.99
shap==0.41.0
simplejson==3.17.6
six==1.16.0
slicer==0.0.7
smart-open==5.2.1
smmap==5.0.0
soundfile==0.12.1
soupsieve==2.3.1
soxr==0.3.5
spacy==3.5.3
spacy-legacy==3.0.12
spacy-loggers==1.0.4
spark-tensorflow-distributor==1.0.0
SQLAlchemy==1.4.39
sqlparse==0.4.2
srsly==2.4.6
ssh-import-id==5.11
stack-data==0.6.2
statsmodels==0.13.2
tabulate==0.8.10
tangled-up-in-unicode==0.2.0
tenacity==8.1.0
tensorboard==2.11.0
tensorboard-data-server==0.6.1
tensorboard-plugin-profile==2.11.2
tensorboard-plugin-wit==1.8.1
tensorflow-cpu==2.11.1
tensorflow-estimator==2.11.0
tensorflow-io-gcs-filesystem==0.32.0
termcolor==2.3.0
terminado==0.13.1
testpath==0.6.0
thinc==8.1.10
threadpoolctl==2.2.0
tiktoken==0.4.0
tokenize-rt==4.2.1
tokenizers==0.13.3
tomli==2.0.1
torch==1.13.1+cpu
torchvision==0.14.1+cpu
tornado==6.1
tqdm==4.64.1
traitlets==5.1.1
transformers==4.29.2
typeguard==2.13.3
typer==0.7.0
typing-inspect==0.9.0
typing_extensions==4.3.0
ujson==5.4.0
unattended-upgrades==0.1
urllib3==1.26.11
virtualenv==20.16.3
visions==0.7.5
wadllib==1.3.6
wasabi==1.1.2
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==0.58.0
Werkzeug==2.0.3
whatthepatch==1.0.2
widgetsnbextension==3.6.1
wordcloud==1.9.2
wrapt==1.14.1
xgboost==1.7.5
xxhash==3.2.0
yapf==0.31.0
yarl==1.9.2
ydata-profiling==4.2.0
zipp==3.8.0

📝 Description of the feature

Add option to import/export snapshot as matrix

An obstacle to quick adoption of TBROMs in a workflow is the need for the user to create additional functions to convert data in tabular format (generally the most common import/export format from other 3D tools) to snapshot vectors and vice versa.
On the input side, there is the potential error of using column-major, rather than row-major vectorisation, while for outputs, the dimensionality is not directly inferable from the vector and needs to retrieve from the TBROM settings anyway.

e.g. n x 3 matrix must be flattened to 3n vector on input. On output, dimensionality must be retrieved from tbrom.field_output_dim and then used to reshape the vector

flattens to

Incorporating these functions simplifies user work and abstracts the snapshot format.

💡 Steps for implementing the feature

A possible option is to provide an additional boolean argument to TbRom.generate_snapshot and TbRom.generate_snapshot to export as a matrix e.g. as_matrix.

Input flattening could use ndarray.flatten(order='C') directly, explicitly using the row-major style. Boolean argument is not strictly necessary, since structure can be inferred from input shape. This would also required the ability to pass inputs in memory (see Issue 107).

Output reshaping can use the numpy.reshape(vec, newshape, order='C') function. New shape columns are retrieved from TBROM field_output_dim, rows calculated from len(vec) / dim.

In both cases, reading/writing matrices to disk need not necessarily be supported. The specific formats required may vary with upstream and downstream programs, so writing can be left to the user.

🔗 Useful links and references

Input matrices would require implementation of Issue #107

Looks like in the latest Python 3.7 support, Python3.11 has been removed from the supported versions. Could we please add it back? Thanks in advance!

🔍 Before submitting the issue

• I have searched among the existing issues
• I am using a Python virtual environment

🐞 Description of the bug

https://nvd.nist.gov/vuln/detail/CVE-2020-1971

📝 Steps to reproduce

pip install pytwin
Run vulnerability scan on resulting venv

💻 Which operating system are you using?

Windows

📀 Which ANSYS version are you using?

none

🐍 Which Python version are you using?

3.10

📦 Installed packages

numpy==1.26.1
pandas==2.1.1
python-dateutil==2.8.2
pytwin==0.5.0
pytz==2023.3.post1
pywin32==306
six==1.16.0
tzdata==2023.3

📝 Description of the feature

Include TBROM info in the TwinRuntime.print_model_info() and expose it in TwinModel so it can be accessed from there.

print_model_info() provides a useful summary, in one place, of almost all twin model information, except for the TBROM information. It would also be useful to have the TBROM info here too.

In addition, the information is still useful at the higher TwinModel level, but is only only accessible through self._twin_runtime.

💡 Steps for implementing the feature

Exposure at high level could be a simple as adding and extra method to TwinModel:

def print_model_info(self, max_var_to_print=np.inf):
    self._twin_runtime.print_model_info(max_var_to_print)

TBROM info print could be done within TwinRuntime, or within TwinModel.

🔗 Useful links and references

No response