Error applying projection

[GLuca74]

Hello,
summarizing, from a third part software I receive an expression that contain a IQueryable<TEntity>, I also receive the TEntity EF Model metadata. This expression may contains any valid query.
Under some circumstances, I have to limit the materialized data, so I check the Model metadata, I choice wich properties I want to retrive from database and I add a projection to the query attaching a select call to the expression.

All seems work but in some cases the query fails despite it is a valid query.

for example, having this model

    public class Nation
    {
        public Guid NationID { get; set; }
        public string NationName { get; set; }
        public virtual ICollection<City> Cities { get; set; }
    }
    public class City
    {
        public Guid CityID { get; set; }
        public virtual Nation Nation { get; set; }
        public string CityName { get; set; }

        public virtual ICollection<District> Districts { get; set; }
    }

    public class District
    {
        public Guid DistrictID { get; set; }
        public string DistrictName { get; set; }
        public virtual City City { get; set; }
    }

when I receive this query
ctx.Set<Nation>().Select(itm => itm.Cities.First().Districts)

I apply the projection and I get the query
ctx.Set<Nation>().Select(itm => itm.Cities.First().Districts).Select(itm => itm.Select(itmSelect => new { DistrictName = itmSelect.DistrictName })).ToArray();
this fails with

System.ArgumentException
  HResult=0x80070057
  Messaggio=Property 'System.String DistrictName' is not defined for type 'System.Collections.Generic.ICollection`1[ConsoleApp1.District]' Arg_ParamName_Name
  Origine=System.Linq.Expressions
  Analisi dello stack:
   in System.Linq.Expressions.Expression.Property(Expression expression, PropertyInfo property)
   in System.Linq.Expressions.Expression.MakeMemberAccess(Expression expression, MemberInfo member)
   in System.Linq.Expressions.MemberExpression.Update(Expression expression)
   in Microsoft.EntityFrameworkCore.Query.ReplacingExpressionVisitor.VisitMember(MemberExpression memberExpression)
   in System.Linq.Expressions.MemberExpression.Accept(ExpressionVisitor visitor)
   in System.Dynamic.Utils.ExpressionVisitorUtils.VisitArguments(ExpressionVisitor visitor, IArgumentProvider nodes)
   in System.Linq.Expressions.ExpressionVisitor.VisitNew(NewExpression node)
   in System.Linq.Expressions.NewExpression.Accept(ExpressionVisitor visitor)
   in Microsoft.EntityFrameworkCore.Query.Internal.NavigationExpandingExpressionVisitor.ProcessSelect(NavigationExpansionExpression source, LambdaExpression selector)
   in Microsoft.EntityFrameworkCore.Query.Internal.NavigationExpandingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   in System.Linq.Expressions.MethodCallExpression.Accept(ExpressionVisitor visitor)
   in System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
   in Microsoft.EntityFrameworkCore.Query.Internal.NavigationExpandingExpressionVisitor.PendingSelectorExpandingExpressionVisitor.Visit(Expression expression)
   in Microsoft.EntityFrameworkCore.Query.Internal.NavigationExpandingExpressionVisitor.Expand(Expression query)
   in Microsoft.EntityFrameworkCore.Query.QueryTranslationPreprocessor.Process(Expression query)
   in Microsoft.EntityFrameworkCore.Query.QueryCompilationContext.CreateQueryExecutor[TResult](Expression query)
   in Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.<>c__DisplayClass9_0`1.<Execute>b__0()
   in Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQuery[TResult](Object cacheKey, Func`1 compiler)
   in Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.Execute[TResult](Expression query)
   in Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable`1.GetEnumerator()
   in System.Collections.Generic.LargeArrayBuilder`1.AddRange(IEnumerable`1 items)
   in System.Collections.Generic.EnumerableHelpers.ToArray[T](IEnumerable`1 source)
   in System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   in ConsoleApp1.Program.Main(String[] args) in C:\EFError\ConsoleApp1\ConsoleApp1\Program.cs: riga 16

Of course the query fails also if I directly write it in a test project(like the attached)

another expample is this query :
ctx.Set<City>().GroupBy(itm => EF.Property<Guid>(itm, "NationID")).Select(itm => itm.First())
after the projection is added
ctx.Set<City>().GroupBy(itm => EF.Property<Guid>(itm, "NationID")).Select(itm => itm.First()).Select(itmSelect=>new {itmSelect.CityName}).ToArray();
this fail with :

System.InvalidOperationException
  HResult=0x80131509
  Messaggio=The LINQ expression 'ProjectionBindingExpression: 0' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to 'AsEnumerable', 'AsAsyncEnumerable', 'ToList', or 'ToListAsync'. See https://go.microsoft.com/fwlink/?linkid=2101038 for more information.
  Origine=Microsoft.EntityFrameworkCore.Relational
  Analisi dello stack:
   in Microsoft.EntityFrameworkCore.Query.Internal.RelationalProjectionBindingExpressionVisitor.Visit(Expression expression)
   in Microsoft.EntityFrameworkCore.Query.Internal.RelationalProjectionBindingExpressionVisitor.VisitMember(MemberExpression memberExpression)
   in System.Linq.Expressions.MemberExpression.Accept(ExpressionVisitor visitor)
   in Microsoft.EntityFrameworkCore.Query.Internal.RelationalProjectionBindingExpressionVisitor.Visit(Expression expression)
   in Microsoft.EntityFrameworkCore.Query.Internal.RelationalProjectionBindingExpressionVisitor.VisitNew(NewExpression newExpression)
   in System.Linq.Expressions.NewExpression.Accept(ExpressionVisitor visitor)
   in Microsoft.EntityFrameworkCore.Query.Internal.RelationalProjectionBindingExpressionVisitor.Visit(Expression expression)
   in Microsoft.EntityFrameworkCore.Query.Internal.RelationalProjectionBindingExpressionVisitor.Translate(SelectExpression selectExpression, Expression expression)
   in Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.TranslateSelect(ShapedQueryExpression source, LambdaExpression selector)
   in Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   in System.Linq.Expressions.MethodCallExpression.Accept(ExpressionVisitor visitor)
   in System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
   in Microsoft.EntityFrameworkCore.Query.QueryCompilationContext.CreateQueryExecutor[TResult](Expression query)
   in Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.<>c__DisplayClass9_0`1.<Execute>b__0()
   in Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQuery[TResult](Object cacheKey, Func`1 compiler)
   in Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.Execute[TResult](Expression query)
   in Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable`1.GetEnumerator()
   in System.Collections.Generic.LargeArrayBuilder`1.AddRange(IEnumerable`1 items)
   in System.Collections.Generic.EnumerableHelpers.ToArray[T](IEnumerable`1 source)
   in System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   in ConsoleApp1.Program.Main(String[] args) in C:\EFError\ConsoleApp1\ConsoleApp1\Program.cs: riga 18

Can you please check where the problem is and advise a workaround?

Many thanks

ConsoleApp1.zip

Provider : Microsoft.EntityFrameworkCore.SqlServer 6.0.12

[roji]

@GLuca74 thanks for the code samples, I can see the errors happening.

Here's a minimal repro for the first case, where a single projection works but when it's broken down to two projections it fails:

Minimal repro for the first case

await using var ctx = new Mycontext();
await ctx.Database.EnsureDeletedAsync();
await ctx.Database.EnsureCreatedAsync();

// WORKS:
_ = ctx.Set<Nation>()
    .Select(n => n.Cities.First().Districts.Select(d => new { d.DistrictName }))
    .ToArray();

// FAILS (same thing broken down to two projections):
var R2 = ctx.Nations
    .Select(n => n.Cities.First().Districts)
    .Select(districts => districts.Select(d => new { d.DistrictName }))
    .ToArray();

public class Mycontext : DbContext
{
    public DbSet<Nation> Nations { get; set; }

    protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        => optionsBuilder
            .UseSqlServer(@"Server=localhost;Database=test;User=SA;Password=Abcd5678;Connect Timeout=60;ConnectRetryCount=0;Encrypt=false")
            .LogTo(Console.WriteLine, LogLevel.Information)
            .EnableSensitiveDataLogging();
}

public class Nation
{
    public Guid NationID { get; set; }
    public string NationName { get; set; }
    public virtual ICollection<City> Cities { get; set; }
}

public class City
{
    public Guid CityID { get; set; }
    public virtual Nation Nation { get; set; }
    public string CityName { get; set; }

    public virtual ICollection<District> Districts { get; set; }
}

public class District
{
    public Guid DistrictID { get; set; }
    public string DistrictName { get; set; }
    public virtual City City { get; set; }
}

For the second case, I can see the error happening with "'ProjectionBindingExpression:0' could not be translated" with 6.0. With 7.0 I instead get the error "The given key 'EmptyProjectionMember' was not present in the dictionary.".

Minimal repro for the second case

await using var ctx = new Mycontext();
await ctx.Database.EnsureDeletedAsync();
await ctx.Database.EnsureCreatedAsync();

var R1 = ctx.Set<City>()
    .GroupBy(itm => EF.Property<Guid>(itm, "NationID"))
    .Select(itm => itm.First())
    .Select(itmSelect => new { itmSelect.CityName })
    .ToArray();

public class Mycontext : DbContext
{
    public DbSet<Nation> Nations { get; set; }

    protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        => optionsBuilder
            .UseSqlServer(@"Server=localhost;Database=test;User=SA;Password=Abcd5678;Connect Timeout=60;ConnectRetryCount=0;Encrypt=false")
            .LogTo(Console.WriteLine, LogLevel.Information)
            .EnableSensitiveDataLogging();
}

public class Nation
{
    public Guid NationID { get; set; }
    public string NationName { get; set; }
    public virtual ICollection<City> Cities { get; set; }
}

public class City
{
    public Guid CityID { get; set; }
    public virtual Nation Nation { get; set; }
    public string CityName { get; set; }

    public virtual ICollection<District> Districts { get; set; }
}

public class District
{
    public Guid DistrictID { get; set; }
    public string DistrictName { get; set; }
    public virtual City City { get; set; }
}

[ajcvickers]

Note from triage: likely related to pending selector work.

[GLuca74]

Hello,

we have to decide if we can keep invest time on wath we are doing now(wrote on top), so I am 'forced' to ask you if this will fix for version 6.x.

Thank you for understanding

[ajcvickers]

@GLuca74 This won't get fixed in a 6.0 release.

[GLuca74]

Of course I can not know the reasons you decided to not fix this in version 6 but honestly this surprice me.
Version 6 is lts and if you not fix bugs like this I wonder why a lts versions should be preferred.

Can you please advide a workaroud?

Many thanks

[roji]

@GLuca74 we do fix bugs in 6.0, but that doesn't mean that all bugs can be fixed in patch versions. Specifically, the fix here is likely to be quite complex, and therefore bringing with it some considerable risk. See https://learn.microsoft.com/en-us/ef/core/what-is-new/release-planning#patch-releases for more information on what we consider eligible for patching.

[GLuca74]

@roji Many thanks for the informations u post, I read the document, can I hope this wil fix in a Minor releases of 6?

[roji]

There isn't going to be a minor release of 6. This will likely be fixed only for 8.0.