Removed deepforest_docs folder

Use models from Huggingface

Clean up the tests

Update style for tests

Switch to opencv-python-headless

Using opencv-python was causing installation conflicts with
albumentations which relies on the headless version. We were only using
the GUI aspects of opencv in a single, debugging only, function. So
this switches that function to matplotlib and updates the dependencies.

Fixes #464.

Don't install opencv via apt for testing

We want to test the Python installs so we shouldn't be doing an apt
install. This is particularly true since we've switched to
opencv-python-headless as the dependency.

.github/workflows/Conda-app.yml

@@ -36,7 +36,6 @@ jobs:
         run: |
           sudo apt-get update
           sudo apt-get install -y libgl1-mesa-glx libegl1-mesa
-          sudo apt-get install -y python3-opencv
 
       - name: Install Conda environment with Micromamba
         uses: mamba-org/setup-micromamba@v1

deepforest/utilities.py

@@ -20,6 +20,8 @@
 from deepforest import _ROOT
 import json
 import urllib.request
+from huggingface_hub import hf_hub_download
+from huggingface_hub.utils._errors import RevisionNotFoundError, HfHubHTTPError
 
 
 def read_config(config_path):
@@ -54,6 +56,50 @@ def update_to(self, b=1, bsize=1, tsize=None):
         self.update(b * bsize - self.n)
 
 
+def fetch_model(save_dir, repo_id, model_filename, version="main"):
+    """Downloads a model from Hugging Face and saves it to a specified
+    directory.
+
+    Parameters:
+    - save_dir (str): The directory where the model will be saved.
+    - repo_id (str): The ID of the Hugging Face repository (e.g., "weecology/deepforest-tree").
+    - model_filename (str): The name of the model file in the repository (e.g., "NEON.pt").
+    - version (str): The version or branch of the model to download (e.g., "main", "v1.0.0"). Default is "main".
+
+    Returns:
+    - output_path (str): The path where the model is saved.
+    """
+    # Ensure the save directory exists
+    os.makedirs(save_dir, exist_ok=True)
+
+    # Define the output path
+    output_path = os.path.join(save_dir, model_filename)
+
+    try:
+        # Download the model from Hugging Face
+        hf_hub_download(
+            repo_id=repo_id,
+            filename=model_filename,
+            local_dir=save_dir,
+            revision=version  # Specify the version or branch of the model to download
+        )
+        print(f"Model saved to: {output_path}")
+    except RevisionNotFoundError as e:
+        print(f"Error: {e}")
+        print(
+            f"Check that the file '{model_filename}' and revision '{version}' exist in the repository '{repo_id}'."
+        )
+    except HfHubHTTPError as e:
+        print(f"HTTP Error: {e}")
+        print(
+            "There might be a problem with your internet connection or the file may not exist."
+        )
+    except Exception as e:
+        print(f"An unexpected error occurred: {e}")
+
+    return version, output_path
+
+
 def use_bird_release(
         save_dir=os.path.join(_ROOT, "data/"), prebuilt_model="bird", check_release=True):
     """
@@ -63,61 +109,10 @@ def use_bird_release(
         prebuilt_model: Currently only accepts "NEON", but could be expanded to include other prebuilt models. The local model will be called prebuilt_model.h5 on disk.
         check_release (logical): whether to check github for a model recent release. In cases where you are hitting the github API rate limit, set to False and any local model will be downloaded. If no model has been downloaded an error will raise.
     Returns: release_tag, output_path (str): path to downloaded model
-
     """
-
-    # Naming based on pre-built model
-    output_path = os.path.join(save_dir, prebuilt_model + ".pt")
-
-    if check_release:
-        # Find latest github tag release from the DeepLidar repo
-        _json = json.loads(
-            urllib.request.urlopen(
-                urllib.request.Request(
-                    'https://api.github.com/repos/Weecology/BirdDetector/releases/latest',
-                    headers={'Accept': 'application/vnd.github.v3+json'},
-                )).read())
-        asset = _json['assets'][0]
-        url = asset['browser_download_url']
-
-        # Check the release tagged locally
-        try:
-            release_txt = pd.read_csv(save_dir + "current_bird_release.csv")
-        except BaseException:
-            release_txt = pd.DataFrame({"current_bird_release": [None]})
-
-        # Download the current release it doesn't exist
-        if not release_txt.current_bird_release[0] == _json["html_url"]:
-
-            print("Downloading model from BirdDetector release {}, see {} for details".
-                  format(_json["tag_name"], _json["html_url"]))
-
-            with DownloadProgressBar(unit='B',
-                                     unit_scale=True,
-                                     miniters=1,
-                                     desc=url.split('/')[-1]) as t:
-                urllib.request.urlretrieve(url,
-                                           filename=output_path,
-                                           reporthook=t.update_to)
-
-            print("Model was downloaded and saved to {}".format(output_path))
-
-            # record the release tag locally
-            release_txt = pd.DataFrame({"current_bird_release": [_json["html_url"]]})
-            release_txt.to_csv(save_dir + "current_bird_release.csv")
-        else:
-            print("Model from BirdDetector Repo release {} was already downloaded. "
-                  "Loading model from file.".format(_json["html_url"]))
-
-        return _json["html_url"], output_path
-    else:
-        try:
-            release_txt = pd.read_csv(save_dir + "current_release.csv")
-        except BaseException:
-            raise ValueError("Check release argument is {}, but no release has been "
-                             "previously downloaded".format(check_release))
-
-        return release_txt.current_release[0], output_path
+    return fetch_model(save_dir,
+                       repo_id="weecology/deepforest-bird",
+                       model_filename="bird.pt")
 
 
 def use_release(
@@ -132,58 +127,9 @@ def use_release(
     Returns: release_tag, output_path (str): path to downloaded model
 
     """
-    # Naming based on pre-built model
-    output_path = os.path.join(save_dir, prebuilt_model + ".pt")
-
-    if check_release:
-        # Find latest github tag release from the DeepLidar repo
-        _json = json.loads(
-            urllib.request.urlopen(
-                urllib.request.Request(
-                    'https://api.github.com/repos/Weecology/DeepForest/releases/latest',
-                    headers={'Accept': 'application/vnd.github.v3+json'},
-                )).read())
-        asset = _json['assets'][0]
-        url = asset['browser_download_url']
-
-        # Check the release tagged locally
-        try:
-            release_txt = pd.read_csv(save_dir + "current_release.csv")
-        except BaseException:
-            release_txt = pd.DataFrame({"current_release": [None]})
-
-        # Download the current release it doesn't exist
-        if not release_txt.current_release[0] == _json["html_url"]:
-
-            print("Downloading model from DeepForest release {}, see {} "
-                  "for details".format(_json["tag_name"], _json["html_url"]))
-
-            with DownloadProgressBar(unit='B',
-                                     unit_scale=True,
-                                     miniters=1,
-                                     desc=url.split('/')[-1]) as t:
-                urllib.request.urlretrieve(url,
-                                           filename=output_path,
-                                           reporthook=t.update_to)
-
-            print("Model was downloaded and saved to {}".format(output_path))
-
-            # record the release tag locally
-            release_txt = pd.DataFrame({"current_release": [_json["html_url"]]})
-            release_txt.to_csv(save_dir + "current_release.csv")
-        else:
-            print("Model from DeepForest release {} was already downloaded. "
-                  "Loading model from file.".format(_json["html_url"]))
-
-        return _json["html_url"], output_path
-    else:
-        try:
-            release_txt = pd.read_csv(save_dir + "current_release.csv")
-        except BaseException:
-            raise ValueError("Check release argument is {}, but no release "
-                             "has been previously downloaded".format(check_release))
-
-        return release_txt.current_release[0], output_path
+    return fetch_model(save_dir,
+                       repo_id="weecology/deepforest-tree",
+                       model_filename="NEON.pt")
 
 
 def read_pascal_voc(xml_path):
@@ -836,4 +782,4 @@ def project_boxes(df, root_dir, transform=True):
     raise NotImplementedError(
         "This function is deprecated. Please use image_to_geo_coordinates instead.")
 
-    return df
+    return df

deepforest/visualize.py

@@ -2,6 +2,7 @@
 import os
 import pandas as pd
 import matplotlib
+from matplotlib import pyplot as plt
 from PIL import Image
 import numpy as np
 import pandas.api.types as ptypes
@@ -31,8 +32,7 @@ def view_dataset(ds, savedir=None, color=None, thickness=1):
     if savedir:
         cv2.imwrite("{}/{}".format(savedir, image_path[0]), image)
     else:
-        cv2.imshow(image)
-        cv2.waitKey(0)
+        plt.imshow(image)
 
 
 def format_geometry(predictions, scores=True):

dev_requirements.txt

@@ -7,14 +7,15 @@ docformatter
 docutils<0.18
 pydata-sphinx-theme
 geopandas
+huggingface_hub
 h5py
 matplotlib
 nbmake
 nbsphinx
 nbqa
 numpy
 numpydoc
-opencv-python>=4.5.4
+opencv-python-headless>=4.5.4
 pandas
 pillow>6.2.0
 pip

docs/Makefile

@@ -5,7 +5,7 @@
 SPHINXOPTS    =
 SPHINXBUILD   = python -msphinx
 SPHINXPROJ    = deepforest
-SOURCEDIR     = source
+SOURCEDIR     = .
 BUILDDIR      = _build
 
 # Put it first so that "make" without argument is like "make help".

docs/source/conf.py → docs/conf.py

@@ -25,7 +25,7 @@
 using [reticulate](https://rstudio.github.io/reticulate/) works.
 
 """
-file_obj = open('deepforest_docs/deepforestr.md', 'w')
+file_obj = open('deepforestr.md', 'w')
 readme_url = 'https://raw.githubusercontent.com/weecology/deepforestr/main/README.md'
 file_obj.write(deepforestr_title)
 
@@ -42,7 +42,7 @@
 
 # Create copy of CONTRIBUTING.md
 contributing_url = "https://raw.githubusercontent.com/weecology/DeepForest/main/CONTRIBUTING.md"
-contributing_source = "../../CONTRIBUTING.md"
+contributing_source = "../CONTRIBUTING.md"
 
 if not os.path.exists(contributing_source):
     with urllib.request.urlopen(contributing_url) as response:
@@ -52,7 +52,7 @@
 
 # reading from file1 and writing to file2
 with open(contributing_source, "r") as file1:
-    with open("deepforest_docs/CONTRIBUTING.md", "w") as file2:
+    with open("CONTRIBUTING.md", "w") as file2:
         file2.write(file1.read())
 
 needs_sphinx = "1.8"
@@ -78,7 +78,7 @@
 templates_path = ['_templates']
 
 # The master toctree document.
-master_doc = 'deepforest_docs/index'
+master_doc = 'index'
 
 # General information about the project.
 project = u'DeepForest'

docs/source/deepforest_docs/data_annotation/annotation.md → docs/data_annotation/annotation.md

@@ -4,12 +4,12 @@ Annotation is the most important part of machine learning projects.  If you aren
 ## How should I annotate images?
 For quick annotations of a few images, we recommend using QGIS or ArcGIS. Either as project or unprojected data. Create a shapefile for each image.
 
-![QGISannotation](../../../../www/QGIS_annotation.png)
+![QGISannotation](../../www/QGIS_annotation.png)
 
 ### Label-studio
 For longer term projects, we recommend [label-studio](https://labelstud.io/) as an annotation platform. It has many useful features and is easy to set up.
 
-![QGISannotation](../../../../www/label_studio.png)
+![QGISannotation](../../www/label_studio.png)
 
 ## Do I need annotate all objects in my image?
 Yes! Object detection models use the non-annotated areas of an image as negative data. We know that it can be difficult to annotate all objects in an image, but non-annotation will cause the model *to ignore* objects that should be treated as positive samples, leading to poor model performance. 

docs/developer_resources/authors.rst

@@ -0,0 +1 @@
+.. include:: ../../AUTHORS.rst

docs/developer_resources/history.rst

@@ -0,0 +1 @@
+.. include:: ../../HISTORY.rst

docs/source/deepforest_docs/introduction/getting_started.md → docs/getting_started/getting_started.md

@@ -21,7 +21,7 @@ img = model.predict_image(path=sample_image_path, return_plot=True)
 plt.imshow(img[:,:,::-1])
 ```
 
-![](../../../../www/getting_started1.png)
+![](../../www/getting_started1.png)
 
 
 ** please note that this video was made before the deepforest-pytorch -> deepforest name change. **

docs/index.rst

@@ -0,0 +1,9 @@
+:orphan:
+
+DeepForest
+========================
+
+.. toctree::
+   :maxdepth: 2
+
+   source/index

docs/source/deepforest_docs/installation_and_setup/prebuilt.md → docs/installation_and_setup/prebuilt.md

@@ -6,7 +6,7 @@ DeepForest has two prebuilt models: Bird Detection and Tree Crown Detection.
 
 The model was initially described in [Remote Sensing](https://www.mdpi.com/2072-4292/11/11/1309) on a single site. The prebuilt model uses a semi-supervised approach in which millions of moderate quality annotations are generated using a LiDAR unsupervised tree detection algorithm, followed by hand-annotations of RGB imagery from select sites. Comparisons among geographic sites were added to [Ecological Informatics](https://www.sciencedirect.com/science/article/pii/S157495412030011X). The model was further improved, and the Python package was released in [Methods in Ecology and Evolution](https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/2041-210X.13472).
 
-![image](../../../../www/MEE_Figure4.png)
+![image](../../www/MEE_Figure4.png)
 
 ### Citation
 > Weinstein, B.G.; Marconi, S.; Bohlman, S.; Zare, A.; White, E. Individual Tree-Crown Detection in RGB Imagery Using Semi-Supervised Deep Learning Neural Networks. Remote Sens. 2019, 11, 1309
@@ -23,15 +23,15 @@ The model was initially described in [Ecological Applications](https://esajourna
  ### Citation
 > Weinstein, B.G., Garner, L., Saccomanno, V.R., Steinkraus, A., Ortega, A., Brush, K., Yenni, G., McKellar, A.E., Converse, R., Lippitt, C.D., Wegmann, A., Holmes, N.D., Edney, A.J., Hart, T., Jessopp, M.J., Clarke, R.H., Marchowski, D., Senyondo, H., Dotson, R., White, E.P., Frederick, P. and Ernest, S.K.M. (2022), A general deep learning model for bird detection in high resolution airborne imagery. Ecological Applications. Accepted Author Manuscript e2694. https://doi-org.lp.hscl.ufl.edu/10.1002/eap.2694
 
-![image](../../../../www/example_predictions_small.png)
+![image](../../www/example_predictions_small.png)
 
 ```
 #Load deepforest model and set bird label
 m = main.deepforest(label_dict={"Bird":0})
 m.use_bird_release()
 ```
 
-![](../../../../www/bird_panel.jpg)
+![](../../www/bird_panel.jpg)
 
 We have created a [GPU colab tutorial](https://colab.research.google.com/drive/1e9_pZM0n_v3MkZpSjVRjm55-LuCE2IYE?usp=sharing
 ) to demonstrate the workflow for using the bird model.

docs/source/deepforest_docs/introduction/index.rst → docs/introduction/index.rst

@@ -8,4 +8,3 @@ Welcome to the introduction section.
    :caption: Contents:
 
    landing
-   getting_started

docs/source/deepforest_docs/introduction/landing.md → docs/introduction/landing.md

@@ -2,7 +2,7 @@
 
 DeepForest is a python package for training and predicting ecological objects in airborne imagery. DeepForest comes with models for immediate use and finetuning. Both are single class modules that can be extended to species classification based on new data. Users can extend these models by annotating and training custom models. 
 
-![](../../../../www/image.png)
+![](../../www/image.png)
 
 ## How does deepforest work?