Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python: postgres memory store #1354

Merged
merged 48 commits into from
Jul 11, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
fffe2ae
boilerplate
cschadewitz May 29, 2023
aea2e15
mem store and init
cschadewitz May 30, 2023
b7389f6
add tests, run pre commit
cschadewitz May 31, 2023
5a095d6
Merge remote-tracking branch 'origin/main' into python_postgres_memor…
cschadewitz May 31, 2023
4f7c383
Merge branch 'main' into python_postgres_memory_store
cschadewitz Jun 6, 2023
75beebf
Merge remote-tracking branch 'origin/main' into python_postgres_memor…
cschadewitz Jun 9, 2023
d4c5c1e
implement new mem record field
cschadewitz Jun 9, 2023
bc631ac
Merge branch 'main' into python_postgres_memory_store
cschadewitz Jun 9, 2023
1c2b964
Merge branch 'main' into python_postgres_memory_store
cschadewitz Jun 11, 2023
f5e0a7b
Merge branch 'main' into python_postgres_memory_store
cschadewitz Jun 12, 2023
32a9c37
Merge branch 'main' into python_postgres_memory_store
cschadewitz Jun 13, 2023
8ec42dd
Merge branch 'main' into python_postgres_memory_store
cschadewitz Jun 19, 2023
1fcb2c3
Merge branch 'main' into python_postgres_memory_store
cschadewitz Jun 21, 2023
320b6e3
Merge branch 'main' into python_postgres_memory_store
cschadewitz Jun 22, 2023
bb6cc7d
Apply small changes based on suggestions
cschadewitz Jun 29, 2023
7b52ac6
Merge branch 'main' into python_postgres_memory_store
cschadewitz Jun 29, 2023
fc858ca
param all sql, move to table level collections
cschadewitz Jul 2, 2023
b7de648
Merge remote-tracking branch 'origin/main' into python_postgres_memor…
cschadewitz Jul 2, 2023
560b839
PR comment fixes
cschadewitz Jul 2, 2023
090b7da
Update python/tests/integration/connectors/memory/test_postgres.py
cschadewitz Jul 2, 2023
b47d4bc
implement default dim, add/fix comments, cleanup
cschadewitz Jul 2, 2023
9da0dcf
add dependencies, more comments
cschadewitz Jul 2, 2023
4dc3339
allow batch to find 0-n record for n keys
cschadewitz Jul 2, 2023
23387c9
move to using local records
cschadewitz Jul 2, 2023
cc8de3c
change env parsing
cschadewitz Jul 2, 2023
0edfb6b
update feature matrix
cschadewitz Jul 2, 2023
4c69cdd
add limitation to dimensionality based on docs
cschadewitz Jul 2, 2023
66ed03e
lint
cschadewitz Jul 2, 2023
c27298d
add schema param
cschadewitz Jul 2, 2023
9e9c96e
class doc strings
cschadewitz Jul 2, 2023
954baa5
lint
cschadewitz Jul 2, 2023
4c0de94
Remove vector extension creation
cschadewitz Jul 2, 2023
c1f2700
Initial stab at handling timezones properly
cschadewitz Jul 2, 2023
50430dc
implement timestamp
cschadewitz Jul 3, 2023
65bbaab
Merge branch 'main' into python_postgres_memory_store
cschadewitz Jul 3, 2023
1332320
Merge remote-tracking branch 'origin/main' into python_postgres_memor…
cschadewitz Jul 5, 2023
c7e419d
Merge branch 'main' into python_postgres_memory_store
cschadewitz Jul 6, 2023
fea47d9
Merge branch 'main' into python_postgres_memory_store
dmytrostruk Jul 7, 2023
3d36df0
Merge branch 'main' into python_postgres_memory_store
shawncal Jul 8, 2023
9bf2f52
pr review changes
cschadewitz Jul 8, 2023
0b15c5b
Merge remote-tracking branch 'origin/main' into python_postgres_memor…
cschadewitz Jul 8, 2023
39a36ca
Merge branch 'python_postgres_memory_store' of https://github.com/csc…
cschadewitz Jul 8, 2023
f2574b4
resolve merge conflict in poetry lock file
awharrison-28 Jul 11, 2023
b067e0b
Merge branch 'main' into python_postgres_memory_store
awharrison-28 Jul 11, 2023
e3c1489
changes for linting checks, changed dependency psycopg-binary to psyc…
awharrison-28 Jul 11, 2023
6193a3f
Merge branch 'python_postgres_memory_store' of https://github.com/csc…
awharrison-28 Jul 11, 2023
8c551f6
prepped for integration tests in CI
awharrison-28 Jul 11, 2023
e1e06ad
Merge branch 'main' into python_postgres_memory_store
awharrison-28 Jul 11, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/python-integration-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ jobs:
OpenAI__ApiKey: ${{ secrets.OPENAI__APIKEY }}
Pinecone__ApiKey: ${{ secrets.PINECONE__APIKEY }}
Pinecone__Environment: ${{ secrets.PINECONE__ENVIRONMENT }}
Postgres__Connectionstr: ${{secrets.POSTGRES__CONNECTIONSTR}}
run: |
cd python
poetry run pytest ./tests/integration
6 changes: 3 additions & 3 deletions FEATURE_MATRIX.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@
| Math Skill | ✅ | ✅ | ✅ | |
| Text Skill | ✅ | ✅ | 🔄 | |
| Time Skill | ✅ | ✅ | ✅ | |
| Wait Skill | ✅ | | ✅ | |
| Wait Skill | ✅ | | ✅ | |

## Planning

Expand All @@ -63,10 +63,10 @@
| Weaviate | ✅ | ✅ | ❌ | Currently supported on Python 3.9+, 3.8 coming soon |
| Chroma | ✅ | ✅ | ❌ | |
| Qdrant | ✅ | ❌ | ❌ | |
| Pinecone | ✅ | | ❌ | |
| Pinecone | ✅ | | ❌ | |
| Milvus | ❌ | ❌ | ❌ | Coming soon |
| Sqlite | ✅ | ❌ | ❌ | Vector optimization requires [sqlite-vss](https://github.com/asg017/sqlite-vss) |
| Postgres | ✅ | | ❌ | Vector optimization requires [pgvector](https://github.com/pgvector/pgvector) |
| Postgres | ✅ | | ❌ | Vector optimization requires [pgvector](https://github.com/pgvector/pgvector) |
| CosmosDB | ✅ | ❌ | ❌ | CosmosDB is not optimized for vector storage |
| Redis | ✅ | ❌ | ❌ | Vector optimization requires [RediSearch](https://redis.io/docs/stack/search) |

Expand Down
1 change: 1 addition & 0 deletions python/.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@ AZURE_OPENAI_ENDPOINT=""
AZURE_OPENAI_API_KEY=""
PINECONE_API_KEY=""
PINECONE_ENVIRONMENT=""
POSTGRES_CONNECTION_STRING=""
982 changes: 595 additions & 387 deletions python/poetry.lock

Large diffs are not rendered by default.

4 changes: 4 additions & 0 deletions python/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,10 @@ weaviate-client = "^3.18.0"
[tool.poetry.group.pinecone.dependencies]
pinecone-client = "^2.2.2"

[tool.poetry.group.postgres.dependencies]
psycopg-pool = "^3.1.7"
psycopg = "^3.1.9"

[tool.isort]
profile = "black"

Expand Down
2 changes: 2 additions & 0 deletions python/semantic_kernel/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,13 +18,15 @@
azure_openai_settings_from_dot_env,
openai_settings_from_dot_env,
pinecone_settings_from_dot_env,
postgres_settings_from_dot_env,
)

__all__ = [
"Kernel",
"NullLogger",
"openai_settings_from_dot_env",
"azure_openai_settings_from_dot_env",
"postgres_settings_from_dot_env",
"pinecone_settings_from_dot_env",
"PromptTemplateConfig",
"PromptTemplate",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@


class PineconeMemoryStore(MemoryStoreBase):
"""A memory store that uses Pinecone as the backend."""

_logger: Logger
_pinecone_api_key: str
_pinecone_environment: str
Expand All @@ -42,8 +44,8 @@ def __init__(
Arguments:
pinecone_api_key {str} -- The Pinecone API key.
pinecone_environment {str} -- The Pinecone environment.
logger {Optional[Logger]} -- The logger to use. (default: {None})
default_dimensionality {int} -- The default dimensionality to use for new collections.
logger {Optional[Logger]} -- The logger to use. (default: {None})
"""
if default_dimensionality > MAX_DIMENSIONALITY:
raise ValueError(
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Copyright (c) Microsoft. All rights reserved.

from semantic_kernel.connectors.memory.postgres.postgres_memory_store import (
cschadewitz marked this conversation as resolved.
Show resolved Hide resolved
PostgresMemoryStore,
)

__all__ = ["PostgresMemoryStore"]
Loading