Unable to reproduce the result in ./examples/cxrmate.ipynb

[passby111]

I download the weight from https://huggingface.co/aehrc/cxrmate. And run ./examples. The result of cxrmate-multi-tf.ipynb is the same. But when I run cxrmate.ipynb, the result is different with that in ipynb.
First study result:
Findings: Frontal and lateral views of the chest were obtained. A large bore dual lumen central venous catheter terminates within the right internal jugular central venous catheter terminates within the right atrium. The lungs are unchanged. The heart size and right internal jugular central venous catheter ends in unchanged. The heart size is unchanged. The heart size is unchanged. The heart size is unchanged. The aorta remains mildly tortuous and the upper thoracic aorta remains mildly tortuous and tortuous and hilar contours are unchanged. The cardiac silhouette size is unchanged. The aorta remains mildly tortuous and tortuous and tortuous and tortuous and tortuous and hilar contours are unchanged. The aorta is unchanged. Increased interstitial abnormality is unchanged. The cardiac silhouette size is unchanged. The aorta is unchanged. The cardiac silhouette size is unchanged. There is unchanged. The aorta is unchanged. The aorta is unchanged. The cardiac silhouette size of the osseous structures are within normal. The aorta is unchanged. The cardiac silhouette is unchanged. The aorta is unchanged. The aorta is unchanged with atherosclerotic calcifications are unchanged. The aorta is unchanged. The aorta is tortuous and the osseous structures are within normal. The aorta calcified and tortuous atherosclerotic calcifications are within normal. The aorta is tortuous and tortuous and tortuous and tortuous. The aorta is calcified tortuous atherosclerotic calcifications are diffusely calcified thoracic aorta calcified and tortuous. The aorta
Impression:

Findings: Frontal and lateral views of the chest were obtained. A large bore dual lumen central venous catheter terminates within the right internal jugular central venous catheter terminates within the right atrium. The lungs are unchanged. The heart size and right internal jugular central venous catheter ends in unchanged. The heart size is unchanged. The heart size is unchanged. The heart size is unchanged. The aorta remains mildly tortuous and the upper thoracic aorta remains mildly tortuous and tortuous and hilar contours are unchanged. The cardiac silhouette size is unchanged. The aorta remains mildly tortuous and tortuous and tortuous and tortuous and tortuous and hilar contours are unchanged. The aorta is unchanged. Increased interstitial abnormality is unchanged. The cardiac silhouette size is unchanged. The aorta is unchanged. The cardiac silhouette size is unchanged. There is unchanged. The aorta is unchanged. The aorta is unchanged. The cardiac silhouette size of the osseous structures are within normal. The aorta is unchanged. The cardiac silhouette is unchanged. The aorta is unchanged. The aorta is unchanged with atherosclerotic calcifications are unchanged. The aorta is unchanged. The aorta is tortuous and the osseous structures are within normal. The aorta calcified and tortuous atherosclerotic calcifications are within normal. The aorta is tortuous and tortuous and tortuous and tortuous. The aorta is calcified tortuous atherosclerotic calcifications are diffusely calcified thoracic aorta calcified and tortuous. The aorta
Impression:

Seconde study result:
Findings: PA and the lungs are within normal. There is moderately tortuous and tortuous and the heart size is moderately tortuous and the lungs are within normal. There is moderately tortuous. There is moderately tortuous and the aortic knob is moderately tortuous. There is moderately tortuous. There is moderately tortuous and tortuous and the aorta is moderately tortuous. There is unchanged. There is a large bore central pulmonary vascularity is moderately tortuous and the aortic knob and tortuous and the aorta is unchanged. There is unchanged. There is unchanged. There is moderately tortuous. There is moderately tortuous. There is moderately tortuous. There is moderately tortuous. The cardiac silhouette is unchanged. Low lung volumes are within normal. The cardiac silhouette is moderately tortuous. There is unchanged. The cardiac silhouette is moderately tortuous. There is moderately tortuous. The cardiac silhouette is moderately tortuous. There is unchanged. There is unchanged. The cardiac silhouette is moderately tortuous and tortuous. Low lung volumes are within normal. There is unchanged. The cardiac silhouette is moderately tortuous. The cardiac silhouette is unchanged. The cardiac silhouette is unchanged. There is moderately tortuous and tortuous. There is unchanged. There is unchanged. There is moderately tortuous. The cardiac silhouette is moderately tortuous and tortuous and the knob calcifications are within normal. There is moderately tortuous and the atherosclerotic calcifications are within normal.
Impression:

Findings: The heart remains moderately enlarged. Low lung volumes are low. Low lung volumes are low. Low lung volumes are low. Low lung volumes are relatively low. Low lung volumes are low. Low lung volumes are low. Low lung volumes are low. Low lung volumes are low. Low lung volumes are low, and there is low, and there is low, and there is grossly clear. Low lung volumes are low, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, and there is low, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however, however,
Impression:

I wonder why? Otherwise, I pip install -r requirements.txt , the version of transformer is 4.42.1, in this version is 4.42, it will report:Traceback (most recent call last):
File "/tmp/cxrmate/examples/cxrmate-multi-tf.py", line 59, in
outputs = encoder_decoder.generate(
File "/root/miniconda3/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/root/miniconda3/lib/python3.8/site-packages/transformers/generation/utils.py", line 1597, in generate
model_kwargs = self._prepare_encoder_decoder_kwargs_for_generation(
File "/root/miniconda3/lib/python3.8/site-packages/transformers/generation/utils.py", line 523, in _prepare_encoder_decoder_kwargs_for_generation
model_kwargs["encoder_outputs"]: ModelOutput = encoder(**encoder_kwargs)
File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
TypeError: forward() got an unexpected keyword argument 'output_attentions'

I changed version to 4.30.1, it will not report an error.
I would like to know your Torch and Transformers versions, directly following the requirements will download the latest version.
I hope to receive your reply! Thank you very much!

[passby111]

Otherwise，I use the weight from https://huggingface.co/aehrc/cxrmate. The test rusult is abnormal

[anicolson]

I’ll have to look into it when I get a chance. I had to update the code to make it compatible with the latest PEFT package, which could of caused some issues… Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Zhuhao Wang ***@***.***> Sent: Saturday, June 8, 2024 8:33:07 PM To: aehrc/cxrmate ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [aehrc/cxrmate] Unable to reproduce the result in ./examples/cxrmate.ipynb (Issue #6) Otherwise，I use the weight from https://huggingface.co/aehrc/cxrmate. The test rusult is abnormal — Reply to this email directly, view it on GitHub<#6 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AGHGZ7WHPN4S2AGS76M5QYDZGLMWHAVCNFSM6AAAAABI4GGNXSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJVHE4DCMRRGE>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[passby111]

I have modified the version to '4.31.0', '2.0.1+cu117', but it still does not work.

[passby111]

I find when I initiaze cxrmate and cxrmate-tf model, it will report Some weights of the model checkpoint at /data1/wangzhuhao/hf_model/hf_model/cxrmate-tf were not used when initializing LongitudinalPromptMultiCXREncoderDecoderModel: ['decoder.base_model.model.bert.encoder.layer.3.attention.self.query.base_layer.weight', 'decoder.base_model.model.bert.encoder.layer.2.attention.self.key.base_layer.weight', 'decoder.base_model.model.bert.encoder.layer.4.attention.self.query.base_layer.weight', 'decoder.base_model.model.bert.encoder.layer.5.attention.self.query.base_layer.weight', 'decoder.base_model.model.bert.encoder.layer.4.attention.self.key.base_layer.weight', 'decoder.base_model.model.bert.encoder.layer.1.attention.self.query.base_layer.weight', 'decoder.base_model.model.bert.encoder.layer.4.attention.self.key.base_layer.bias', 'decoder.base_model.model.bert.encoder.layer.1.attention.self.query.base_layer.bias', 'decoder.base_model.model.bert.encoder.layer.2.attention.self.query.base_layer.bias', 'decoder.base_model.model.bert.encoder.layer.1.attention.self.key.base_layer.bias', 'decoder.base_model.model.bert.encoder.layer.3.attention.self.key.base_layer.bias', 'decoder.base_model.model.bert.encoder.layer.5.attention.self.query.base_layer.bias', 'decoder.base_model.model.bert.encoder.layer.5.attention.self.key.base_layer.bias', 'decoder.base_model.model.bert.encoder.layer.2.attention.self.query.base_layer.weight', 'decoder.base_model.model.bert.encoder.layer.1.attention.self.key.base_layer.weight', 'decoder.base_model.model.bert.encoder.layer.0.attention.self.key.base_layer.weight', 'decoder.base_model.model.bert.encoder.layer.4.attention.self.query.base_layer.bias', 'decoder.base_model.model.bert.encoder.layer.0.attention.self.key.base_layer.bias', 'decoder.base_model.model.bert.encoder.layer.0.attention.self.query.base_layer.weight', 'decoder.base_model.model.bert.encoder.layer.3.attention.self.query.base_layer.bias', 'decoder.base_model.model.bert.encoder.layer.2.attention.self.key.base_layer.bias', 'decoder.base_model.model.bert.encoder.layer.0.attention.self.query.base_layer.bias', 'decoder.base_model.model.bert.encoder.layer.5.attention.self.key.base_layer.weight', 'decoder.base_model.model.bert.encoder.layer.3.attention.self.key.base_layer.weight']

• This IS expected if you are initializing LongitudinalPromptMultiCXREncoderDecoderModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
• This IS NOT expected if you are initializing LongitudinalPromptMultiCXREncoderDecoderModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
  Some weights of LongitudinalPromptMultiCXREncoderDecoderModel were not initialized from the model checkpoint at /data1/wangzhuhao/hf_model/hf_model/cxrmate-tf and are newly initialized: ['decoder.base_model.model.bert.encoder.layer.3.attention.self.query.weight', 'decoder.base_model.model.bert.encoder.layer.2.attention.self.query.weight', 'decoder.base_model.model.bert.encoder.layer.1.attention.self.key.weight', 'decoder.base_model.model.bert.encoder.layer.2.attention.self.query.bias', 'decoder.base_model.model.bert.encoder.layer.5.attention.self.query.weight', 'decoder.base_model.model.bert.encoder.layer.3.attention.self.key.bias', 'decoder.base_model.model.bert.encoder.layer.5.attention.self.key.bias', 'decoder.base_model.model.bert.encoder.layer.0.attention.self.key.bias', 'decoder.base_model.model.bert.encoder.layer.1.attention.self.query.bias', 'decoder.base_model.model.bert.encoder.layer.0.attention.self.query.bias', 'decoder.base_model.model.bert.encoder.layer.1.attention.self.query.weight', 'decoder.base_model.model.bert.encoder.layer.2.attention.self.key.bias', 'decoder.base_model.model.bert.encoder.layer.3.attention.self.query.bias', 'decoder.base_model.model.bert.encoder.layer.0.attention.self.key.weight', 'decoder.base_model.model.bert.encoder.layer.5.attention.self.key.weight', 'decoder.base_model.model.bert.encoder.layer.4.attention.self.query.weight', 'decoder.base_model.model.bert.encoder.layer.4.attention.self.query.bias', 'decoder.base_model.model.bert.encoder.layer.1.attention.self.key.bias', 'decoder.base_model.model.bert.encoder.layer.4.attention.self.key.weight', 'decoder.base_model.model.bert.encoder.layer.3.attention.self.key.weight', 'decoder.base_model.model.bert.encoder.layer.0.attention.self.query.weight', 'decoder.base_model.model.bert.encoder.layer.4.attention.self.key.bias', 'decoder.base_model.model.bert.encoder.layer.2.attention.self.key.weight', 'decoder.base_model.model.bert.encoder.layer.5.attention.self.query.bias']

And when I initiaze cxrmate-single-tf and cxrmate-multi-tf, it won't report. I guess there may be an issue with the uploaded weights

[anicolson]

Thanks @passby111,

I am going to look at this as soon as I can. I currently have a deadline I have to meet (EMNLP), which makes it tricky for me to fix this issue immediately. But as soon as I have a little bit of time, I can fix this issue. I do have the original weights, so I assume this issue can be fixed.

Thank you for your patience.

[passby111]

I found that it is possible that when using Lora to save weights, the name of the base layer will be changed, which is manifested as an additional. base_layer. I used the following code to modify the key name of the weight file and can now reproduce your results. Thank you for your reply amidst your busy schedule. Wishing you a smooth submission!
for key, value in weights.items():
new_key = key.replace(".base_layer", "")
new_weights[new_key] = value

[passby111]

Hi, https://huggingface.co/aehrc/cxrmate-gt This link is invalid, Can you give this weight again when you have time. Thanks.

[anicolson]

Hi @passby111,

I finally had some time to work on this :).

I have implemented some quick fixes and updated the requirements.

Seems to be working okay. But will test and check a few more things.

This is the env I tested with (reflective of the new requirements.txt):

pip list

``` Package Version ----------------------------- ------------------------ absl-py 2.0.0 accelerate 0.30.1 addict 2.4.0 aiohttp 3.9.0 aiosignal 1.3.1 alabaster 0.7.13 albumentations 1.4.7 alembic 1.12.0 altgraph 0.17.4 amqp 5.1.1 annotated-types 0.5.0 antlr4-python3-runtime 4.9.3 anyio 4.2.0 anytree 2.9.0 appdirs 1.4.4 argcomplete 3.2.3 argon2-cffi 23.1.0 argon2-cffi-bindings 21.2.0 arrow 1.3.0 astor 0.8.1 astroid 3.0.0 astropy 5.3.3 asttokens 2.4.0 astunparse 1.6.3 async-lru 2.0.4 atom 0.10.2 attrs 23.1.0 Automat 22.10.0 av 12.0.0 Babel 2.12.1 backcall 0.2.0 bcrypt 4.0.1 beartype 0.18.5 beautifulsoup4 4.12.2 beniget 0.4.1 bidict 0.22.1 billiard 4.1.0 binaryornot 0.4.4 biopandas 0.4.1 biopython 1.83 black 23.9.1 bleach 6.0.0 blessed 1.20.0 blinker 1.6.2 blis 0.9.1 bokeh 3.2.2 boto3 1.34.14 botocore 1.34.14 Bottleneck 1.3.7 bpytop 1.0.68 branca 0.6.0 bravado 11.0.3 bravado-core 6.1.1 brewer2mpl 1.4.1 Brotli 1.1.0 bs4 0.0.1 build 1.0.3 bytecode 0.15.0 cachetools 5.3.3 cadquery 2.3.0 cairocffi 1.6.1 calmsize 0.1.3 casa-formats-io 0.2.1 catalogue 2.0.10 catboost 1.2.3 cbook 0.0.4 ccimport 0.4.2 celery 5.3.4 certifi 2023.7.22 cffi 1.16.0 cftime 1.6.2 chardet 5.2.0 charset-normalizer 3.3.0 click 8.1.7 click-didyoumean 0.3.0 click-plugins 1.1.1 click-repl 0.3.0 cligj 0.7.2 cloudpathlib 0.16.0 cloudpickle 2.2.1 cmaes 0.10.0 colorama 0.4.6 colorcet 3.0.1 colorlog 6.7.0 comm 0.1.4 commonmark 0.9.1 confection 0.1.4 ConfigArgParse 1.7 configmypy 0.1.0 configobj 5.0.8 configparser 6.0.0 constantly 15.1.0 contextlib2 21.6.0 contourpy 1.1.1 cookiecutter 2.4.0 corenlp 0.0.14 cov-core 1.15.0 coverage 7.3.2 croniter 2.0.1 cryptography 41.0.4 cumm-cu122 0.5.2 cupy 13.0.0b1 cycler 0.12.0 cymem 2.0.8 Cython 3.0.2 dash 2.16.1 dash-core-components 2.0.0 dash-html-components 2.0.0 dash-table 5.0.0 dask 2023.9.3 databricks-cli 0.18.0 datasets 2.15.0 dateutils 0.6.12 deap 1.4.1 debugpy 1.8.0 decorator 5.1.1 deepdiff 6.7.1 deepspeed 0.13.5 defusedxml 0.7.1 denoising-diffusion-pytorch 2.0.6 Deprecated 1.2.14 descartes 1.1.0 dgl 2.2 dicom 0.9.9.post1 diff-gaussian-rasterization 0.0.0 dill 0.3.7 distlib 0.3.7 dlhpcstarter 0.1.7 dlx 1.0.4 dm-haiku 0.0.10 dm-tree 0.1.8 dnspython 2.4.2 docker 6.1.3 docker-pycreds 0.4.0 docopt 0.6.2 docplex 2.25.236 docstring-parser 0.15 docutils 0.20.1 dparse 0.6.3 duckdb 0.9.3.dev2267 dulwich 0.21.6 e3nn 0.5.1 easydict 1.10 echo 0.8.0 editor 1.6.5 einops 0.7.0 ema-pytorch 0.4.8 embreex 2.17.7.post3 emoji 2.9.0 enaml 0.16.1 entrypoints 0.4 et-xmlfile 1.1.0 evaluate 0.4.2 exceptiongroup 1.2.0 executing 2.0.0 ez-setup 0.9 fast-histogram 0.11 fastapi 0.108.0 fastdtw 0.3.4 fasteners 0.19 fastjsonschema 2.18.1 fastrlock 0.8.2 fdb 2.0.2 filelock 3.12.4 fiona 1.9.6 fire 0.5.0 flann 1.9.2 flann 1.9.2 flash-attention 1.0.0 Flask 3.0.0 Flask-SocketAPI 0.2 Flask-SocketIO 5.3.6 Flask-SQLAlchemy 3.1.1 Flask-WTF 1.2.1 flatbuffers 23.5.26 fonttools 4.43.0 fqdn 1.5.1 freetype-py 2.4.0 frozendict 2.3.8 frozenlist 1.4.0 fsspec 2023.9.2 ftfy 6.1.1 future 0.18.3 futures 3.1.1 fvcore 0.1.5.post20221221 gast 0.5.4 GDAL 3.7.2 gdown 4.7.1 gensim 4.3.2 geographiclib 2.0 geopandas 0.14.3 geopy 2.4.0 gevent 23.9.1 ggplot 0.11.5 gitdb 4.0.10 GitPython 3.1.37 glue-core 1.13.1 glue-qt 0.2.0 glue-vispy-viewers 1.1.0 glueviz 1.2.0 gmsh 4.12.2 goatools 1.3.11 google-auth 2.23.3 google-auth-oauthlib 1.0.0 google-pasta 0.2.0 GPUtil 1.4.0 gpxpy 1.5.0 graphviz 0.20.3 greenlet 3.0.0 GridDataFormats 1.0.2 grpcio 1.59.0 grpcio-tools 1.59.0 gunicorn 21.2.0 gzip-reader 0.1 h11 0.14.0 h3 3.7.6 h5py 3.10.0 hjson 3.1.0 holoviews 1.17.1 hsluv 5.0.4 html5lib 1.1 httpcore 1.0.4 httplib2 0.22.0 httpx 0.27.0 hugedict 2.12.10 huggingface-hub 0.23.4 humanfriendly 10.0 husl 4.0.3 hydra-core 1.3.2 hyperlink 21.0.0 hyperopt 0.2.7 hypothesis 6.87.1 icalendar 5.0.10 icecream 2.1.3 idna 3.4 igraph 0.10.8 ihm 1.0 imagecodecs 2024.1.1 imageio 2.31.6 imagesize 1.4.1 importlib-metadata 6.8.0 incremental 22.10.0 inflate64 0.3.1 inflection 0.5.1 iniconfig 2.0.0 inquirer 3.2.1 interval 1.0.0 iopath 0.1.10 ipadic 1.0.0 ipykernel 6.25.2 ipyleaflet 0.17.4 ipyparallel 8.7.0 ipython 8.22.2 ipython-genutils 0.2.0 ipywidgets 8.1.1 isodate 0.6.1 isoduration 20.11.0 isort 5.12.0 itsdangerous 2.1.2 jaraco.classes 3.3.0 jax 0.4.18 jaxlib 0.4.18+cuda12.cudnn89 jedi 0.19.1 jeepney 0.8.0 Jinja2 3.1.2 jmespath 1.0.1 jmp 0.0.4 joblib 1.3.2 johnnydep 1.20.3 json5 0.9.24 jsonpickle 3.0.2 jsonpointer 2.4 jsonref 1.1.0 jsonschema 4.19.1 jsonschema-specifications 2023.7.1 jupyter_client 8.6.1 jupyter_core 5.7.2 jupyter-events 0.10.0 jupyter-lsp 2.2.4 jupyter_server 2.13.0 jupyter_server_terminals 0.5.3 jupyterlab 4.1.5 jupyterlab_pygments 0.3.0 jupyterlab_server 2.25.4 jupyterlab_templates 0.5.2 jupyterlab_widgets 3.0.10 kappabenchmark 0.0.10 kappaconfig 1.0.31 kappadata 1.4.13 kappamodules 0.1.61 kappaprofiler 1.0.11 kappaschedules 0.0.31 kappautils 0.0.7 keras 2.14.0 keras-nightly 2.15.0.dev2023092207 Keras-Preprocessing 1.1.2 keyring 24.2.0 kiwisolver 1.4.5 kombu 5.3.2 kornia 0.7.2 kornia_rs 0.1.3 langcodes 3.3.0 lark 1.1.9 lazy 1.6 lazy_loader 0.3 libmagic 1.0 lightning 2.3.0 lightning-cloud 0.5.57 lightning-utilities 0.10.0 linkify-it-py 2.0.2 lit 17.0.6 llvmlite 0.42.0 lmdb 1.4.1 locket 1.0.0 loguru 0.7.2 lxml 4.9.3 Mako 1.2.4 mapbox-earcut 1.0.1 Markdown 3.4.4 markdown-it-py 3.0.0 MarkupSafe 2.1.3 matplotlib 3.8.0 matplotlib-inline 0.1.6 matplotlib-venn 0.11.9 mccabe 0.7.0 mda-xdrlib 0.2.0 MDAnalysis 2.7.0 mdit-py-plugins 0.4.0 mdtraj 1.10.0 mdurl 0.1.2 mecab-python3 1.0.9 meshio 5.3.5 meson 1.2.2 MinkowskiEngine 0.5.4 mistune 3.0.2 ml-collections 0.1.1 ml-dtypes 0.3.1 mlflow 2.8.1 mmtf-python 1.1.3 mock 5.1.0 modelcif 0.9 monai 1.3.0+57.gd7137cf4.dirty monotonic 1.6 more-itertools 10.1.0 moss 0.5 mpl-scatter-density 0.7 mpmath 1.3.0 mrcfile 1.5.0 msgpack 1.0.7 multidict 6.0.4 multiprocess 0.70.15 multitasking 0.0.11 multivolumefile 0.2.3 munch 4.0.0 murmurhash 1.0.10 mypy-extensions 1.0.0 natsort 8.4.0 nbclassic 1.0.0 nbclient 0.10.0 nbconvert 7.16.3 nbformat 5.10.3 neptune 1.8.6 nest-asyncio 1.5.8 netCDF4 1.6.4 networkx 3.1 neuraloperator 0.3.0 nflows 0.14 nglview 3.1.2 nh3 0.2.14 nibabel 5.1.0 ninja 1.11.1 nltk 3.8.1 nodeenv 1.8.0 normflows 1.7.3 nose 1.3.7 nose-cov 1.6 nose-exclude 0.5.0 nose-parameterized 0.6.0 nose-xunitmp 0.4.1 nosexcover 1.0.11 notebook 7.1.2 notebook_shim 0.2.4 numexpr 2.8.7 numpy 1.26.4 numpy-stl 3.0.1 nvidia-ml-py 12.550.52 nvitop 1.3.3.dev8+gc2f7af4 oauthlib 3.2.2 obonet 1.0.0 omegaconf 2.3.0 opacus 1.4.1 opencv-python-headless 4.9.0.80 OpenMM 8.0.0 openpyxl 3.1.2 opt-einsum 3.3.0 opt-einsum-fx 0.1.4 optuna 3.3.0 ordered-set 4.1.0 orjson 3.9.15 overrides 7.7.0 oyaml 1.0 packaging 23.2 pandas 2.1.1 pandas-flavor 0.6.0 pandocfilters 1.5.1 panel 1.2.3 param 1.13.0 paramiko 3.3.1 parse 1.19.1 parso 0.8.3 partd 1.4.1 pathos 0.3.1 pathspec 0.11.2 pathy 0.10.3 patsy 0.5.3 Paver 1.3.4 pbr 5.11.1 pccm 0.4.11 pdbfixer 1.9.0 peewee 3.16.3 peft 0.11.1 pegen 0.2.0 pexpect 4.8.0 pickleshare 0.7.5 Pillow 10.0.1 pingouin 0.5.4 pip 24.0 pipreqs 0.4.13 pipx 1.4.3 pkgconfig 1.5.5 pkginfo 1.9.6 platformdirs 3.11.0 plotly 5.20.0 pluggy 1.3.0 ply 3.11 plyfile 1.0.3 pooch 1.7.0 portalocker 2.8.2 pox 0.3.3 ppft 1.7.6.7 preshed 3.0.9 prettyprint 0.1.5 prettytable 3.9.0 ProDy 2.4.1 prometheus_client 0.20.0 prompt-toolkit 3.0.43 protobuf 4.25.1 psutil 5.9.8 ptyprocess 0.7.0 pure-eval 0.2.2 pvextractor 0.3 py-cpuinfo 9.0.0 py-grit 0.0.1a2 py-spy 0.4.0.dev2 py4j 0.10.9.7 py7zr 0.20.6 pyarrow 14.0.1 pyarrow-hotfix 0.6 pyasn1 0.5.0 pyasn1-modules 0.3.0 pybcj 1.0.1 pybind11 2.11.1 pycairo 1.25.0 pycocoevalcap 1.2 pycocotools 2.0.8 pycollada 0.7.2 pycosat 0.6.3 pycparser 2.21 pycryptodomex 3.19.0 pyct 0.5.0 pydantic 2.7.1 pydantic_core 2.18.2 pydeck 0.8.0 pydicom 2.4.3 pydot 2.0.0 pyenchant 3.2.2 pyerfa 2.0.0.3 pyflann 1.6.13 PyGithub 2.1.1 Pygments 2.16.1 pyinstaller 6.7.0 pyinstaller-hooks-contrib 2024.6 PyJWT 2.8.0 pyliblzfse 0.4.1 pylint 3.0.0 pymesh 1.0.2 pymeshlab 2023.12.post1 pymongo 4.5.0 pymultinest 2.12 PyNaCl 1.5.0 pynvml 11.5.0 pyopencl 2023.1.2 PyOpenGL 3.1.7 pyparsing 3.1.1 PyPDF2 3.0.1 pyppmd 1.0.0 pyproj 3.6.1 pyproject_hooks 1.0.0 pyproject-metadata 0.7.1 PyQt5 5.15.9 PyQt5-Qt5 5.15.2 PyQt5-sip 12.12.2 PyQt6 6.5.2 PyQt6-Qt6 6.5.2 PyQt6-sip 13.5.2 pyqtgraph 0.13.7 pyquaternion 0.9.9 pyrsistent 0.20.0 PySocks 1.7.1 pyspnego 0.10.1 pytest 7.4.2 pytest-mock 3.14.0 python-constraint 1.4.0 python-dateutil 2.8.2 python-docx 1.1.0 python-engineio 4.7.1 python-etcd 0.4.5 python-igraph 0.10.8 python-json-logger 2.0.7 python-multipart 0.0.6 python-prctl 1.8.1 python-slugify 8.0.1 python-socketio 5.9.0 python-utils 3.8.1 pythran 0.14.0 pytools 2023.1.1 pytorch-concurrent-dataloader 0.0.7 pytorch-fid 0.3.0 pytorch-lightning 2.1.2 pytz 2023.3.post1 pyviz_comms 3.0.0 PyYAML 6.0.1 pyzmq 25.1.1 pyzstd 0.15.9 qimage2ndarray 1.10.0 qtconsole 5.4.4 QtPy 2.4.0 Quandl 3.7.0 querystring-parser 1.2.4 radgraph 0.0.9 radio-beam 0.3.4 rdflib 7.0.0 rdkit 2023.9.1 readchar 4.0.5 readme-renderer 42.0 recommonmark 0.7.1 referencing 0.30.2 regex 2023.10.3 reportlab 4.0.5 requests 2.31.0 requests-ntlm 1.2.0 requests-oauthlib 1.3.1 requests-toolbelt 1.0.0 retrying 1.3.4 retworkx 0.13.1 rfc3339-validator 0.1.4 rfc3986 2.0.0 rfc3986-validator 0.1.1 rfc3987 1.3.8 rich 13.6.0 rocksdb 0.9.2 rpds-py 0.10.3 rsa 4.9 Rtree 1.2.0 ruamel.yaml 0.17.35 ruamel.yaml.clib 0.2.8 runs 1.2.0 rustworkx 0.13.1 s3transfer 0.10.0 safetensors 0.4.3 safety 2.3.5 scikit-fuzzy 0.4.2 scikit-image 0.22.0 scikit-learn 1.5.0 scipy 1.11.4 se3-transformer 1.2.0 seaborn 0.13.0 SecretStorage 3.3.3 Send2Trash 1.8.2 sentencepiece 0.2.0 sentry-sdk 1.44.1 setproctitle 1.3.3 setuptools 68.2.2 shapely 2.0.2 shtab 1.6.4 simple-kNN 1.1.6 simple-websocket 0.10.1 simplejson 3.19.1 sip 6.7.11 six 1.16.0 sklearn-pandas 2.2.0 sklearn2pmml 0.99.0 skorch 0.15.0 slint 1.6.0a8 smart-open 6.4.0 smmap 5.0.1 sniffio 1.3.0 snowballstemmer 2.2.0 sortedcontainers 2.4.0 soundfile 0.12.1 soupsieve 2.5 spacy 3.7.2 spacy-legacy 3.0.12 spacy-loggers 1.0.5 SPARQLWrapper 2.0.0 spectral-cube 0.6.2 Sphinx 7.2.6 sphinx-markdown-tables 0.0.17 sphinxcontrib-applehelp 1.0.7 sphinxcontrib-devhelp 1.0.5 sphinxcontrib-htmlhelp 2.0.4 sphinxcontrib-jsmath 1.0.1 sphinxcontrib-qthelp 1.0.6 sphinxcontrib-serializinghtml 1.1.9 sphinxcontrib-spelling 8.0.0 splines 0.3.1 SQLAlchemy 2.0.21 sqlparse 0.4.4 srsly 2.4.8 stack-data 0.6.3 starlette 0.32.0.post1 statsmodels 0.14.0 stevedore 5.1.0 structlog 23.1.0 stsci.tools 4.1.0 submitit 1.4.6 svg.path 6.3 svgwrite 1.4.3 swagger-spec-validator 3.0.3 sympy 1.12 tabulate 0.9.0 tenacity 8.2.3 tensorboard 2.14.1 tensorboard-data-server 0.7.1 tensorboardX 2.6.2.2 tensorflow-estimator 2.15.0 tensorflow-hub 0.16.1 tensorflow-probability 0.24.0 tensorly 0.8.1 tensorly-torch 0.4.0 termcolor 2.4.0 terminado 0.18.1 text-unidecode 1.3 texttable 1.6.7 tf-slim 1.1.0 thinc 8.2.2 threadpoolctl 3.2.0 tifffile 2023.9.26 timer4 1.1.0 timezonefinder 6.2.0 timm 0.9.16 tinycss2 1.2.1 tinycudann 1.7 tokenizers 0.19.1 toml 0.10.2 tomli 2.0.1 tomlkit 0.12.1 toolshed 0.4.6 toolz 0.12.0 topaz-em 0.2.5 torch 2.1.1 torch-cluster 1.6.2 torch-geometric 2.3.1 torch-harmonics 0.6.5 torch-scatter 2.1.2 torch-sparse 0.6.18 torch-spline-conv 1.2.2 torchaudio 2.1.0 torchdata 0.7.1 torchinfo 1.8.0 torchmetrics 1.2.0 torchtyping 0.1.4 torchvision 0.16.0 tornado 6.3.3 tqdm 4.66.1 traitlets 5.14.2 traittypes 0.2.1 transformers 4.41.2 trimesh 3.23.5 triton 2.1.0 twine 4.0.2 Twisted 23.8.0 typed-argument-parser 1.8.1 typeguard 4.1.5 typer 0.9.0 types-python-dateutil 2.8.19.14 typing_extensions 4.12.0 typing-inspect 0.9.0 tyro 0.5.12 tzdata 2023.3 uc-micro-py 1.0.2 uri-template 1.3.0 urllib3 2.0.6 userpath 1.9.2 uvicorn 0.25.0 versioneer 0.29 vine 5.0.0 virtualenv 20.24.5 virtualenv-clone 0.5.7 virtualenvwrapper 4.8.4 viser 0.1.7 vispy 0.14.0 wandb 0.16.6 wasabi 1.1.2 wcwidth 0.2.8 weasel 0.3.4 webcolors 1.13 webencodings 0.5.1 websocket 0.2.1 websocket-client 1.6.4 websockets 12.0 Werkzeug 3.0.0 wfdb 4.1.2 wheel 0.41.2 widgetsnbextension 4.0.10 wimpy 0.6 wrapt 1.15.0 wsproto 1.2.0 WTForms 3.0.1 xarray 2024.2.0 xgboost 2.0.0 xlrd 2.0.1 XlsxWriter 3.2.0 xmltodict 0.13.0 xmod 1.8.1 xray 0.7.0 xxhash 3.4.1 xyzservices 2023.7.0 yacs 0.1.8 yapf 0.40.2 yarg 0.1.9 yarl 1.9.2 yfinance 0.2.30 yolk 0.4.3 yourdfpy 0.0.53 zipp 3.17.0 zombie-imp 0.0.2 zope.event 5.0 zope.interface 6.0 zstandard 0.22.0 ```

Here are the results I got with config/test_huggingface/longitudinal_gen_prompt_cxr-bert.yaml:

test_report_cxr-bert: 0.7043785452842712
test_report_chexbert_f1_macro: 0.3780640065670013

I am working to get a few more metrics working (we just migrated to a new cluster, so a few things are not working)

The scores will vary between runs, so it's not deterministic (there will be variability in the scores). I suspect it is the metrics, as even with the same reports, they sometimes give different scores (this needs to be tested and confirmed). This is an issue that I have not pinpointed yet). But those are som

Will look at it a bit more in the next couple of days and check a bunch of things.

Thanks for pointing all this out! Its a great help!

[passby111]

Hi, thank for your efforts! I found your test result of config/test_huggingface/longitudinal_gt_prompt_tf.yaml will change due to the batch size setting. This doesn't seem very reasonable

[anicolson]

Okay, I’ll have to investigate that. Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Zhuhao Wang ***@***.***> Sent: Wednesday, June 19, 2024 3:05:19 PM To: aehrc/cxrmate ***@***.***> Cc: Nicolson, Aaron (H&B, Herston) ***@***.***>; Comment ***@***.***> Subject: Re: [aehrc/cxrmate] Unable to reproduce the result in ./examples/cxrmate.ipynb (Issue #6) Hi, thank for your efforts! I found your test result of config/test_huggingface/longitudinal_gt_prompt_tf.yaml will change due to the batch size setting. This doesn't seem very reasonable — Reply to this email directly, view it on GitHub<#6 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AGHGZ7XR3SZEHSWVTR44VQ3ZIEGQ7AVCNFSM6AAAAABI4GGNXSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZXG42TCNZVGQ>. You are receiving this because you commented.Message ID: ***@***.***>

[yihp]

Hi, thank for your efforts! I found your test result of config/test_huggingface/longitudinal_gt_prompt_tf.yaml will change due to the batch size setting. This doesn't seem very reasonable

Hello,
May I ask what is the best batch size setting in your training test?