Data are missing after filtering

[ghost]

Hello guys!

Firstly, i want to really thank you for your work!.

I have a big dataset which looks like that:

I apply filtering at certain LAT and LON to choose the area (geo window) which i am interested. The filtering is done using dictionary like this:

list_of_values = [23. , 23.05, 23.1 , 23.15, 23.2 , 23.25, 23.3 , 23.35, 23.4 ,
23.45, 23.5 , 23.55, 23.6 , 23.65, 23.7 , 23.75, 23.8 , 23.85,
23.9 , 23.95, 24. , 24.05, 24.1 , 24.15, 24.2 , 24.25, 24.3 ,
24.35, 24.4 , 24.45, 24.5 , 24.55, 24.6 , 24.65, 24.7 , 24.75,
24.8 , 24.85, 24.9 , 24.95, 25. , 25.05, 25.1 , 25.15, 25.2 ,
25.25, 25.3 , 25.35, 25.4 , 25.45, 25.5 , 25.55, 25.6 , 25.65,
25.7 , 25.75, 25.8 , 25.85, 25.9 , 25.95, 26. , 26.05, 26.1 ,
26.15, 26.2 , 26.25, 26.3 , 26.35, 26.4 , 26.45, 26.5 , 26.55,
26.6 , 26.65, 26.7 , 26.75, 26.8 , 26.85, 26.9 , 26.95, 27. ,
27.05, 27.1 , 27.15, 27.2 , 27.25, 27.3 , 27.35, 27.4 , 27.45,
27.5]

vaex_df_sparse_crete = vaex_df_sparse[vaex_df_sparse.LON.isin(list_of_values)]

list_of_values = [34. , 34.05, 34.1 , 34.15, 34.2 , 34.25, 34.3 , 34.35, 34.4 ,
34.45, 34.5 , 34.55, 34.6 , 34.65, 34.7 , 34.75, 34.8 , 34.85,
34.9 , 34.95, 35. , 35.05, 35.1 , 35.15, 35.2 , 35.25, 35.3 ,
35.35, 35.4 , 35.45, 35.5 , 35.55, 35.6 , 35.65, 35.7 , 35.75,
35.8 , 35.85, 35.9 , 35.95, 36. , 36.05, 36.1 , 36.15, 36.2 ,
36.25, 36.3 , 36.35, 36.4 , 36.45, 36.5]

vaex_df_sparse_crete = vaex_df_sparse_crete[vaex_df_sparse_crete.LAT.isin(list_of_values)]

After the filtering is implemented i am seeing that some dates (hourly sampling) are missing which is pretty weird because the data are provided by a software. Do you see anything wrong in the way i am treating the dataset;;

Is there a better way to apply filtering;;

Do you think that maybe after saving the filtered dataset an error is happening;

Kind Regards,

GK

[NickCrews]

It looks like you could keep your filtering simpler?

min_lat = 34.0
max_lat = 36.5
min_lon = 23.0
max_lon = 27.5

filtered = df[
   (df.LAT >= min_lat) &
   (df.LAT <= max_lat) &
   (df.LON >= min_lon) &
   (df.LON <= max_lon) &
]

Why don't you do the filtering, and from there you can see which rows indices are looking abnormal, and then you can go back to the original unfiltered data and find those rows and see if they look weird before?

[JovanVeljanoski]

@GKengineering

Hm.. what you are describing should not happen. Is it possible for you to give us a reproducible example (something we can run on our end in case this is a bug). It does not have to be the original data - fake values is nice, just to illustrate what you mean.

Although at a quick glance I don't see anything wrong with your approach.

A quick way to check.. is to do the opposite filter (using the ~ operator) and see if those suspect values re-appear.

[ghost]

Jovan, @NickCrews thanks for the help and the quick reply!
Kind Regards
GK