Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix swift package + add example swift implementation of batched api #3562

Merged
merged 1 commit into from
Oct 11, 2023

Conversation

zshannon
Copy link
Contributor

@zshannon zshannon commented Oct 9, 2023

hey there, thanks for an amazing repo!

the swift package stopped compiling for me and it was tough to debug because the automated tests here are passing, so thought it made sense to add an example use in swift to surface the compiler error and make it trackable as part of the repo. plus, a few people have asked for swift examples with the batch api.

  1. the compiler bug is fixed by including ggml-backed.c in the SPM sources, so that's the change in Package.swift. you can verify it's needed if you like by commenting out the line and running make in examples/swift.

  2. i wrote a rough line for line copy of examples/batched.cpp in swift, though i don't know c++ very well so relied heavily on copilot.. seems to produce the same output but ymmv :)

@jhen0409
Copy link
Sponsor Collaborator

jhen0409 commented Oct 9, 2023

I think we need figure out why CI is still passing after #3548 merged, it seems like the xcodebuild will ignore sources and build the llama scheme directly, and the sources are used when llama is a dependency. If we can't figured it out, I think adding the swift example build to CI is also an option. Great work btw!

UPDATE: Since it includes ggml-backend.h it will build successfully in CI, so that should be expected behavior. However, undefined symbol error will occur when referencing externally if not add ggml-backend.c.

@zshannon
Copy link
Contributor Author

zshannon commented Oct 9, 2023

Derp something is wrong with my swift string parsing, doesn't work for emojis

@ggerganov
Copy link
Owner

ggerganov commented Oct 10, 2023

Thanks for the contribution! Is this example going to be built by the CI?
If not, we should look into adding build support, otherwise it will get obsoleted with API changes in the future

@zshannon
Copy link
Contributor Author

rebased master + added the build step to the root makefile when running on macOS so the CI will pick it up.

examples/swift/Package.swift Outdated Show resolved Hide resolved
Makefile Outdated Show resolved Hide resolved
Makefile Outdated Show resolved Hide resolved
@zshannon
Copy link
Contributor Author

Ok, I'll add a patched GitHub action with those changes

@zshannon zshannon force-pushed the failing-swift-test branch 2 times, most recently from cf00916 to 44e5a51 Compare October 11, 2023 05:51
…ed & build it in ci to validate SPM is correctly configured for dependencies
Copy link
Sponsor Collaborator

@jhen0409 jhen0409 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! 👍

@jhen0409 jhen0409 merged commit 24ba3d8 into ggerganov:master Oct 11, 2023
34 of 39 checks passed
joelkuiper added a commit to vortext/llama.cpp that referenced this pull request Oct 12, 2023
…example

* 'master' of github.com:ggerganov/llama.cpp: (34 commits)
  examples: support LLaVA v1.5 (multimodal model) (ggerganov#3436)
  docs : fix typo GOMP_CPU_AFFINITY (ggerganov#3597)
  cmake : fix add_compile_options on macOS
  typo : it is `--n-gpu-layers` not `--gpu-layers` (ggerganov#3592)
  ci : check if there is enough VRAM (ggerganov#3596)
  server : add completion mode (no chat) (ggerganov#3582)
  prompts : add mnemonics.txt
  server : fix kv cache management (ggerganov#3588)
  main : fix session loading bug (ggerganov#3400)
  server : add parameter -tb N, --threads-batch N (ggerganov#3584)
  common : fix mirostat state when using multiple sequences (ggerganov#3543)
  batched : add bench tool (ggerganov#3545)
  examples : add batched.swift + improve CI for swift (ggerganov#3562)
  Add MPT model to supported models in README.md (ggerganov#3574)
  Minor improvements in GPT2 tokenizer (ggerganov#3567)
  readme : add bloom (ggerganov#3570)
  llm : add bloom models (ggerganov#3553)
  swift : improvements and fixes (ggerganov#3564)
  llm : add MPT support (ggerganov#3417)
  infill. : fix tokenization (ggerganov#3508)
  ...
@zshannon zshannon deleted the failing-swift-test branch October 17, 2023 17:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants