Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding grammar for v0.10.0 #66

Merged
merged 49 commits into from
Nov 19, 2019
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
6126433
Adding grammar for v0.10.0
Oct 27, 2019
b1c7e49
Allowing multiple grammar variants for the same version
Oct 27, 2019
e4203ac
using special grammar variant for the parser
Oct 27, 2019
f4ffdb0
using '.lark' extensions for the grammar files
Oct 27, 2019
04dabd8
fixing warning
Oct 27, 2019
8c48815
add more examples for testing the filter parser
Oct 27, 2019
62a811a
cleanup
Oct 27, 2019
296dbc9
creating a transformer for debugging
Oct 28, 2019
dd86a45
force to keep arguments for operator
Oct 28, 2019
4a5d135
adding rule for strings and numbers
Oct 28, 2019
ae1ad6b
separate int and float in the rule for numbers
Oct 28, 2019
e0c426c
cleanup
Oct 28, 2019
1e2b38e
combining tokens to reduce complexity
Oct 28, 2019
7644b58
first working filter example
Oct 28, 2019
7d8312a
local tests + combining multiple ANDs and ORs
Oct 28, 2019
913c626
cleanup
Oct 29, 2019
f8f077a
update version of lark-parser + fix test
Oct 29, 2019
572b468
adding tests
Oct 29, 2019
43f6844
fix precedence for the test case
Oct 29, 2019
bbaed20
fixing precedence by parentheses
Oct 29, 2019
415443a
installing python package in order
Oct 29, 2019
06dd3fe
formal definition of all rules (some of them just raise an NotImpleme…
Oct 29, 2019
5a41bc5
skeleton class for the transformer
Oct 29, 2019
465ec26
define targets for the tests + cleanup
Oct 29, 2019
a37691c
typo
Oct 29, 2019
9cf3e7b
boost code coverage
Oct 29, 2019
70c03b2
allowing lower-case characters only in the definition of identifiers
Oct 29, 2019
483939d
redefining operator as tokens instead of rules
Oct 30, 2019
0e47883
revere operators when the property isd on the right
Oct 31, 2019
7ba7769
Merge branch 'master' into filter_v0.10.0
fekad Nov 4, 2019
22ba26a
Merge branch 'master' into filter_v0.10.0
CasperWA Nov 8, 2019
e3656c7
Merge branch 'master' into filter_v0.10.0
CasperWA Nov 8, 2019
7349643
allowing white characters in strings
Nov 10, 2019
12fd19c
Merge branch 'master' into filter_v0.10.0
ml-evs Nov 11, 2019
fa9f849
Merge branch 'master' into filter_v0.10.0
ml-evs Nov 11, 2019
b9396e4
using repetition instead of recursion
Nov 12, 2019
d15330f
using the new parser and transformer
Nov 12, 2019
d1999a4
Update optimade/server/entry_collections.py
fekad Nov 14, 2019
ac9e3df
Update optimade/filterparser/tests/test_filterparser.py
fekad Nov 14, 2019
6e9c1ea
Update optimade/filterparser/tests/test_filterparser.py
fekad Nov 14, 2019
195a630
cleanup
Nov 14, 2019
17b8e68
optimized test class
Nov 14, 2019
3f092de
Added filter integration tests with example server
ml-evs Nov 15, 2019
85e9ba5
Skip some tests for unimplemented features
ml-evs Nov 15, 2019
c352196
fix type mismatch warning
Nov 15, 2019
46af4fa
bugfix for the HAS filter
Nov 15, 2019
78fbc39
fixing page_filter issue
Nov 15, 2019
b599d80
quick dirty fix for "deep" queries
Nov 15, 2019
cc6685b
minimal implementation of LENGTH filter
Nov 16, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions optimade/filterparser/__init__.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
__all__ = ["LarkParser", "ParserError"]

from .lark_parser import LarkParser, ParserError

__all__ = [LarkParser, ParserError]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To make this more dynamic, you could do a * import and specify the __all__ in lark_parser.py.
This __all__ then becomes lark_parser.__all__.
In this way, if you decide to reveal other classes from lark_parser, you'll add them to that file's __all__ instead of the __init__.py.

See, e.g., here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Personally, I don't like to usage of * in imports. There are a lot of articles in favour and against its usage (this is just the first match on google). It is similar to the usage of global variable.
Of course, I will modify it as you suggested to keep the repo consistent.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand. We should at some point have a consensus concerning this package of what we do.

12 changes: 4 additions & 8 deletions optimade/filterparser/lark_parser.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,10 @@ class ParserError(Exception):

def get_versions():
dct = defaultdict(dict)
for filename in Path(__file__).parent.joinpath('../grammar').glob('*.lark'):
tags = filename.stem.lstrip('v').split('.')
for filename in Path(__file__).parent.joinpath("../grammar").glob("*.lark"):
tags = filename.stem.lstrip("v").split(".")
version = tuple(map(int, tags[:3]))
variant = 'default' if len(tags) == 3 else tags[-1]
variant = "default" if len(tags) == 3 else tags[-1]
dct[version][variant] = filename
return dct

Expand All @@ -21,7 +21,7 @@ def get_versions():


class LarkParser:
def __init__(self, version=None, variant='default'):
def __init__(self, version=None, variant="default"):

version = version if version else max(available_parsers.keys())

Expand Down Expand Up @@ -53,7 +53,3 @@ def __repr__(self):
return self.tree.pretty()
else:
return repr(self.lark)


if __name__ == '__main__':
print(available_parsers)
253 changes: 165 additions & 88 deletions optimade/filterparser/tests/test_filterparser.py

Large diffs are not rendered by default.

130 changes: 79 additions & 51 deletions optimade/filtertransformers/debug.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,11 @@


class DebugTransformer(Transformer): # pragma: no cover

def __init__(self):
super().__init__()

def __default__(self, data, children, meta):
print('Node: ', data, children)
print("Node: ", data, children)
return data


Expand All @@ -19,135 +18,164 @@ def __init__(self):

def filter(self, arg):
# filter: expression*
print('Node: ', 'filter', arg)
return 'filter'
print("Node: ", "filter", arg)
return "filter"

def constant(self, arg):
# constant: string | number
print('Node: ', 'constant', arg)
return 'constant'
print("Node: ", "constant", arg)
return "constant"

def value(self, arg):
# value: string | number | property
print('Node: ', 'value', arg)
return 'value'
print("Node: ", "value", arg)
return "value"

def value_list(self, arg):
# value_list: [ OPERATOR ] value ( "," [ OPERATOR ] value )*
print('Node: ', 'value_list', arg)
return 'value_list'
print("Node: ", "value_list", arg)
return "value_list"

def value_zip(self, arg):
# value_zip: [ OPERATOR ] value ":" [ OPERATOR ] value (":" [ OPERATOR ] value)*
print('Node: ', 'value_zip', arg)
return 'value_zip'
print("Node: ", "value_zip", arg)
return "value_zip"

def value_zip_list(self, arg):
# value_zip_list: value_zip ( "," value_zip )*
print('Node: ', 'value_zip_list', arg)
return 'value_zip_list'
print("Node: ", "value_zip_list", arg)
return "value_zip_list"

def expression(self, arg):
# expression: expression_clause [ OR expression ]
print('Node: ', 'expression', arg)
return 'expression'
print("Node: ", "expression", arg)
return "expression"

def expression_clause(self, arg):
# expression_clause: expression_phrase [ AND expression_clause ]
print('Node: ', 'expression_clause', arg)
return 'expression_clause'
print("Node: ", "expression_clause", arg)
return "expression_clause"

def expression_phrase(self, arg):
# expression_phrase: [ NOT ] ( comparison | predicate_comparison | "(" expression ")" )
print('Node: ', 'expression_phrase', arg)
return 'expression_phrase'
print("Node: ", "expression_phrase", arg)
return "expression_phrase"

def comparison(self, arg):
# comparison: constant_first_comparison | property_first_comparison
# Note: Do nothing!
print('Node: ', 'comparison', arg)
return 'comparison'
print("Node: ", "comparison", arg)
return "comparison"

def property_first_comparison(self, arg):
# property_first_comparison: property ( value_op_rhs | known_op_rhs | fuzzy_string_op_rhs | set_op_rhs |
# set_zip_op_rhs )
print('Node: ', 'property_first_comparison', arg)
return 'property_first_comparison'
print("Node: ", "property_first_comparison", arg)
return "property_first_comparison"

def constant_first_comparison(self, arg):
# constant_first_comparison: constant value_op_rhs
print('Node: ', 'constant_first_comparison', arg)
return 'constant_first_comparison'
print("Node: ", "constant_first_comparison", arg)
return "constant_first_comparison"

def predicate_comparison(self, arg):
# predicate_comparison: length_comparison
print('Node: ', 'predicate_comparison', arg)
return 'predicate_comparison'
print("Node: ", "predicate_comparison", arg)
return "predicate_comparison"

def value_op_rhs(self, arg):
# value_op_rhs: OPERATOR value
print('Node: ', 'value_op_rhs', arg)
return 'value_op_rhs'
print("Node: ", "value_op_rhs", arg)
return "value_op_rhs"

def known_op_rhs(self, arg):
# known_op_rhs: IS ( KNOWN | UNKNOWN )
print('Node: ', 'known_op_rhs', arg)
return 'known_op_rhs'
print("Node: ", "known_op_rhs", arg)
return "known_op_rhs"

def fuzzy_string_op_rhs(self, arg):
# fuzzy_string_op_rhs: CONTAINS string | STARTS [ WITH ] string | ENDS [ WITH ] string
print('Node: ', 'fuzzy_string_op_rhs', arg)
return 'fuzzy_string_op_rhs'
print("Node: ", "fuzzy_string_op_rhs", arg)
return "fuzzy_string_op_rhs"

def set_op_rhs(self, arg):
# set_op_rhs: HAS ( [ OPERATOR ] value | ALL value_list | ANY value_list | ONLY value_list )
print('Node: ', 'set_op_rhs', arg)
return 'set_op_rhs'
print("Node: ", "set_op_rhs", arg)
return "set_op_rhs"

def set_zip_op_rhs(self, arg):
# set_zip_op_rhs: property_zip_addon HAS ( value_zip | ONLY value_zip_list | ALL value_zip_list |
# ANY value_zip_list )
print('Node: ', 'set_zip_op_rhs', arg)
return 'set_zip_op_rhs'
print("Node: ", "set_zip_op_rhs", arg)
return "set_zip_op_rhs"

def length_comparison(self, arg):
# length_comparison: LENGTH property OPERATOR value
print('Node: ', 'length_comparison', arg)
return 'length_comparison'
print("Node: ", "length_comparison", arg)
return "length_comparison"

def property_zip_addon(self, arg):
# property_zip_addon: ":" property (":" property)*
print('Node: ', 'property_zip_addon', arg)
return 'property_zip_addon'
print("Node: ", "property_zip_addon", arg)
return "property_zip_addon"

def property(self, arg):
# property: IDENTIFIER ( "." IDENTIFIER )*
print('Node: ', 'property', arg)
return 'property'
print("Node: ", "property", arg)
return "property"

def string(self, arg):
# string: ESCAPED_STRING
print('Node: ', 'string', arg)
return 'string'
print("Node: ", "string", arg)
return "string"

def number(self, arg):
# number: SIGNED_INT | SIGNED_FLOAT
print('Node: ', 'number', arg)
return 'number'
print("Node: ", "number", arg)
return "number"

def __default__(self, data, children, meta):
print('Node: ', data, children)
print("Node: ", data, children)
return data


if __name__ == '__main__': # pragma: no cover
if __name__ == "__main__": # pragma: no cover
from optimade.filterparser import LarkParser

p = LarkParser(version=(0, 10, 0))
# t = DebugTransformer()
t = TransformerSkeleton()

f = 'a.a = "text" OR a<a AND NOT b>=8'
# f = 'a.a = "text" OR a<a AND NOT b>=8'

# single list

f = "list HAS < 3"

f = "list HAS < 3, > 4" # -> error
f = "list HAS ALL < 3, > 4"

# multiple lists

f = "list1:list2 HAS < 3 : > 4"
f = "list1:list2 HAS ALL < 3 : > 4"

f = "list1:list2 HAS < 3 : > 4, < 2 : > 5" # -> error
f = "list1:list2 HAS ALL < 3 : > 4, < 2 : > 5"
f = "list1:list2 HAS ALL < 3, < 2 : > 4, > 5" # -> error

# f = 'list1:list2 HAS < 3, > 4' # -> error
# f = 'list1:list2 HAS ALL < 3, > 4' # -> error

f = 'elements:elements_ratios HAS ALL "Al":>0.3333, "Al":<0.3334'
f = 'elements:elements_ratios HAS ALL "Al":>0.3333 AND elements_ratio<0.3334'
f = 'elements:elements_ratios HAS ALL "Al" : >0.3333, <0.3334' # -> error

f = "list1:list2 HAS ALL < 3 : > 4, < 2 : > 5 : > 4, < 2 : > 5" # valid but wrong
f = "ghf.flk<gh" # valid but wrong

# f = ''

tree = p.parse(f)
print(tree)
print(tree.pretty())
Expand Down
Loading