[Feature Request] Support for pipx

[fsacer]

Feature Request or Bug or Another
Feature Request

Describe the feature request or bug or other
Currently theHarvester does not play well with pipx which uses setuptools entry_points for installation. Setuptools themselves recommend moving dependencies to setup.cfg or pyproject.toml file as described in https://setuptools.pypa.io/en/latest/userguide/quickstart.html. This also refers to #1234.

To Reproduce

[2023-05-23 08:45 @ ~]$ pipx install git+https://github.com/laramies/theHarvester
  installed package theHarvester 4.3.0, installed using Python 3.9.2
  These apps are now globally available
    - restfulHarvest
    - theHarvester
done! ✨ 🌟 ✨
[2023-05-23 08:46 @ ~]$ theHarvester -d google.com
Traceback (most recent call last):
  File "/home/user/.local/bin/theHarvester", line 5, in <module>
    from theHarvester import __main__
  File "/home/user/.local/pipx/venvs/theharvester/lib/python3.9/site-packages/theHarvester/__main__.py", line 4, in <module>
    from theHarvester.discovery import dnssearch, takeover, shodansearch
  File "/home/user/.local/pipx/venvs/theharvester/lib/python3.9/site-packages/theHarvester/discovery/dnssearch.py", line 14, in <module>
    from aiodns import DNSResolver
ModuleNotFoundError: No module named 'aiodns'

Expected behaviour
The program should not have module errors after install with pipx.

Screenshots

System Information (System that tool is running on):

• installed package theHarvester 4.3.0, installed using Python 3.9.2
• pipx 1.2.0
• Parrot OS 5.3

Additional context

[NotoriousRebel]

Thank you for the suggestion, I have only used pipx once but was quite impressed, I will see what needs to be done and have it work with pipx. Would it be possible to have both a setup.cfg file and a setup.py file or should we just get rid of the setup.py file altogther and port it over to a setup.cfg file or pyproject.toml file?

[branchvincent]

Homebrew would also find this useful to automatically keep our requirements up to date (see Homebrew/homebrew-core#136408).

I'm happy to work on this, my advice would be to move directly to pyproject.toml since setup.cfg will soon be deprecated pypa/setuptools#3214.

A took a look and I think the only complication is the current use of data_files , which is also deprecated (https://setuptools.pypa.io/en/latest/references/keywords.html, pypa/setuptools#2833). Are these mainly user config files? If so, we could stop installing these and instead have theHarvester search for them under the user's home directory (and also create them if necessary). What do you think?

[NotoriousRebel]

I will have to look more into it this weekend, I believe Kali by default contains the wordlists and maybe we just need to create a function that before main is run to determine if the data files exist and if not just pull them officially from GitHub via prompting the user asking if they want to install them. Only a slim subset of functionality is actually dependent on those data_files and after reading why data_files were removed it does make sense.

Doing some further reading it seems that data_files are still actually supported just deprecated via setuptools and they are already in their own directory so may actually be easier than I thought. @L1ghtn1ng any thoughts on this? If we can integrate data_files properly it should be fine or just pulling down the files externally.

[L1ghtn1ng]

We would need to do data_files properly as don’t want packaging to get broken in kali On 31 Jul 2023, at 17:13, Matt ***@***.***> wrote:  I will have to look more into it this weekend, I believe Kali by default contains the wordlists and maybe we just need to create a function that before main is run to determine if the data files exist and if not just pull them officially from GitHub via prompting the user asking if they want to install them. Only a slim subset of functionality is actually dependent on those data_files and after reading why data_files<pypa/setuptools#2648> were removed it does make sense. Doing some further reading it seems that data_files are still actually supported<https://setuptools.pypa.io/en/latest/userguide/datafiles.html> just deprecated via setuptools and they are already in their own directory so may actually be easier than I thought. @L1ghtn1ng<https://github.com/L1ghtn1ng> any thoughts on this? If we can integrate data_files properly it should be fine or just pulling down the files externally. — Reply to this email directly, view it on GitHub<#1435 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AA3V2QVULWDIAFHD6JGYH5DXS7KRDANCNFSM6AAAAAAYLNE7Q4>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[branchvincent]

Alright created #1505, should be functional and also packages wordlists alongside the code (this makes sense to me as long as its static data and not user-editable).

Only TODO is to handle api-keys.yaml / proxies.yaml. As above, I don't think we should package user config but instead read from ~/.theHarvester/proxies.yaml or similar. Let me know what would work best and I can implement that too.

[L1ghtn1ng]

Without those 2 files included it will not be merged, as those files are integral to how theHarvester works. Also it's a packaging thing as well, as this would break kali currently

[L1ghtn1ng]

Ah okay, there is a lot more that needs sorting as things are going to break all over the place

[branchvincent]

Sure thing, updated the PR to also package the default for those 2 files. They're still read from/etc and /usr/local/etc if found, but otherwise they read from ~/.theHarvester (copying from the defaults if the file doesn't exist, as setuptools recommends)

things are going to break all over the place

Not sure what part is breaking but happy to make sure this change is as smooth as possible for you

[L1ghtn1ng]

Can you put the version code back how it was please, also you removing the requirements dir and files how would you install using just a normal virtualenv, also please put the bin dir back as well as reverting the code you added into main. On 4 Aug 2023, at 17:18, Branch Vincent ***@***.***> wrote:  Sure thing, updated the PR to also package the default for those 2 files. They're still read from/etc and /usr/local/etc if found, but otherwise they read from ~/.theHarvester (copying from the defaults if the file doesn't exist, as setuptools recommends<https://github.com/pypa/setuptools/blob/77c14a8e345a615c749fbac6345b07172cfc1e11/docs/userguide/datafiles.rst?plain=1#L514>) things are going to break all over the place Not sure what part is breaking but happy to make sure this change is as smooth as possible for you — Reply to this email directly, view it on GitHub<#1435 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AA3V2QQX5NE7SSYYNCBIKATXTUOFLANCNFSM6AAAAAAYLNE7Q4>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[branchvincent]

Can you put the version code back how it was please

Can do, I moved it because the simple backend flit reads it from there (since this is a typical place for a package's version string). But I can change to a different backend that allows configuring the location.

removing the requirements dir and files

Yea they are now defined in the dependencies and optional-dependencies tables

how would you install using just a normal virtualenv

Similar to the current workflow, just instead of pip install -r requirements/dev.txt it becomes pip install -e '.[dev]' (-e for editable mode, . for the current directory, and [dev] as an extra to also install the optional dev dependencies). This installs all requirements previously in requirements/ and scripts previously in bin/:

$ pwd
/tmp/theHarvester
$ python3 -m venv .venv
$ . .venv/bin/activate
$ pip install -e '.[dev]'
...
Successfully installed theHarvester-4.4.1
$ python3 -c 'import theHarvester; print(theHarvester.__file__)'
/tmp/theHarvester/theHarvester/__init__.py
$ which theHarvester restfulHarvest
/tmp/theHarvester/.venv/bin/theHarvester
/tmp/theHarvester/.venv/bin/restfulHarvest
$ cat .venv/bin/theHarvester
#!/tmp/theHarvester/.venv/bin/python3.11
# -*- coding: utf-8 -*-
import re
import sys
from theHarvester.__main__ import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

also please put the bin dir back

Can do if this is useful to you on its own, but as mentioned above its not needed since pip now creates/installs the bin scripts for you

as well as reverting the code you added into main

The [project.scripts] table needs a callable function of the form theHarvester.module:function_to_call. If you don't want it in __main__.py, where should I put it?

---

Note if it's easier for you, feel free to leave inline review comments on the PR

[branchvincent]

hey @L1ghtn1ng, just checking if you saw my comments/questions above

[L1ghtn1ng]

@branchvincent yeah, sorry forgot about it, theHarvester.py are wrappers that call the function in main.py so it would need to go in there(theHarvester.py) But in general I would like the least invasive amount of changes as possible to be done as it makes reviewing hard and dont want maintenance to be complicated either