Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exception when creating new migration after updating from 6.0-rc.1 to 6.0-rc.2 or 6.0-rtm (daily) #26405

Closed
schuettecarsten opened this issue Oct 19, 2021 · 40 comments
Labels
area-migrations closed-fixed The issue has been fixed and is/will be included in the release indicated by the issue milestone. customer-reported regression Servicing-approved type-bug
Milestone

Comments

@schuettecarsten
Copy link

schuettecarsten commented Oct 19, 2021

I've updated my test solution from net6.0-rc.1 and efcore6.0-rc.1 to rc.2 versions. Then I try to add a new migration that drops a table from model class CloudManagementTool.DataAccess.Model.Supplier.SupInvoiceReconciliationFile and should create a new table from model class CloudManagementTool.DataAccess.Model.Supplier.SupInvoice.

When running the Add-Migration command from Package Manager Console, I get the following exception:

System.ArgumentException: An item with the same key has already been added. Key: [0, Property: CloudManagementTool.DataAccess.Model.Supplier.SupInvoiceReconciliationFile (Dictionary<string, object>).TenantID (no field, long) Indexer Required FK Index]
   at System.Collections.Generic.TreeSet`1.AddIfNotPresent(T item)
   at Microsoft.EntityFrameworkCore.Migrations.Internal.MigrationsModelDiffer.GetSortedProperties(IEntityType entityType, ITable table)
   at Microsoft.EntityFrameworkCore.Migrations.Internal.MigrationsModelDiffer.GetSortedColumns(ITable table)
   at Microsoft.EntityFrameworkCore.Migrations.Internal.MigrationsModelDiffer.Add(ITable target, DiffContext diffContext)+MoveNext()
   at Microsoft.EntityFrameworkCore.Migrations.Internal.MigrationsModelDiffer.DiffCollection[T](IEnumerable`1 sources, IEnumerable`1 targets, DiffContext diffContext, Func`4 diff, Func`3 add, Func`3 remove, Func`4[] predicates)+MoveNext()
   at System.Linq.Enumerable.ConcatIterator`1.MoveNext()
   at Microsoft.EntityFrameworkCore.Migrations.Internal.MigrationsModelDiffer.Sort(IEnumerable`1 operations, DiffContext diffContext)
   at Microsoft.EntityFrameworkCore.Migrations.Internal.MigrationsModelDiffer.GetDifferences(IRelationalModel source, IRelationalModel target)
   at Microsoft.EntityFrameworkCore.Migrations.Design.MigrationsScaffolder.ScaffoldMigration(String migrationName, String rootNamespace, String subNamespace, String language)
   at Microsoft.EntityFrameworkCore.Design.Internal.MigrationsOperations.AddMigration(String name, String outputDir, String contextType, String namespace)
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.AddMigrationImpl(String name, String outputDir, String contextType, String namespace)
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.AddMigration.<>c__DisplayClass0_0.<.ctor>b__0()
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.OperationBase.<>c__DisplayClass3_0`1.<Execute>b__0()
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.OperationBase.Execute(Action action)
An item with the same key has already been added. Key: [0, Property: CloudManagementTool.DataAccess.Model.Supplier.SupInvoiceReconciliationFile (Dictionary<string, object>).TenantID (no field, long) Indexer Required FK Index]

As the class from the error no longer exists in the code, I have no idea how to fix or workaround this error.

I cannot find any error in the existing migrations, in the class for the new table and in the existing code. The previous code worked as expected. Then I downgraded netcore6.0 and efcore6.0 back to rc.1 and using that version, adding a new migration works without any issues.

PM> Add-Migration RefactorSupplier
Build started...
Build succeeded.
An operation was scaffolded that may result in the loss of data. Please review the migration for accuracy.
To undo this action, use Remove-Migration.

Working EF Core Product Version: 6.0.0-rc.1.21452.10
Failing EF Core Product Version: 6.0.0-rc.2.21480.5

@ajcvickers
Copy link
Member

@schuettecarsten Please try this with the latest daily build. If you are still running into errors, then please attach a small, runnable project or post a small, runnable code listing that reproduces what you are seeing so that we can investigate.

@schuettecarsten
Copy link
Author

It seems to work with 6.0.0-rtm.21519.8

@ajcvickers ajcvickers added closed-fixed The issue has been fixed and is/will be included in the release indicated by the issue milestone. type-bug area-migrations labels Oct 21, 2021
@ajcvickers ajcvickers added this to the 6.0.0 milestone Oct 21, 2021
@schuettecarsten
Copy link
Author

schuettecarsten commented Oct 21, 2021

@ajcvickers - Woohoo, sorry, the issue is still there with latest rtm version, but it seems to be somewhat specific to my context snapshot and this table (see below). The table was removed from my code and should be dropped by the migration, so no class is available any more.

            modelBuilder.Entity("CloudManagementTool.DataAccess.Model.Supplier.SupInvoiceReconciliationFile", b =>
                {
                    b.Property<long>("ID")
                        .ValueGeneratedOnAdd()
                        .HasColumnType("bigint")
                        .HasColumnName("PkID")
                        .HasAnnotation("Neusta:DefaultColumnName", "ID");

                    SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property<long>("ID"), 1L, 1);

                    b.Property<string>("FileName")
                        .HasMaxLength(255)
                        .HasColumnType("nvarchar(255)");

                    b.Property<DateTime>("ImportDate")
                        .HasColumnType("datetime2");

                    b.Property<bool>("IsValid")
                        .HasColumnType("bit");

                    b.Property<byte[]>("RowVersion")
                        .IsConcurrencyToken()
                        .IsRequired()
                        .ValueGeneratedOnAddOrUpdate()
                        .HasColumnType("rowversion");

                    b.Property<long>("SupplierID")
                        .HasColumnType("bigint")
                        .HasColumnName("FkSupplierID");

                    b.Property<long>("TenantID")
                        .HasColumnType("bigint")
                        .HasColumnName("FkTenantID");

                    b.HasKey("ID");

                    b.HasIndex("SupplierID")
                        .HasDatabaseName("IX_SupInvoiceReconciliationFile_FkSupplierID");

                    b.HasIndex("TenantID", "SupplierID", "FileName")
                        .IsUnique()
                        .HasDatabaseName("UQ_SupInvoiceReconciliationFile_FkTenantID_FkSupplierID_FileName")
                        .HasFilter("[FileName] IS NOT NULL")
                        .HasAnnotation("Neusta:TenantFilter:TenantPropertyAdded", null);

                    b.ToTable("InvoiceReconciliationFile", "sup");

                    b.HasAnnotation("Neusta:TenantFilter:TenantIDProperyName", "TenantID");
                });

And somewhere later in the file:

            modelBuilder.Entity("CloudManagementTool.DataAccess.Model.Supplier.SupInvoiceReconciliationFile", b =>
                {
                    b.HasOne("CloudManagementTool.DataAccess.Model.Supplier.SupSupplier", "Supplier")
                        .WithMany()
                        .HasForeignKey("SupplierID")
                        .OnDelete(DeleteBehavior.Restrict)
                        .IsRequired()
                        .HasConstraintName("FK_InvoiceReconciliationFile_Supplier_FkSupplierID");

                    b.HasOne("Neusta.Shared.DataAccess.EntityFramework.TenantFilter.Entities.TenantEntity", "Tenant")
                        .WithMany()
                        .HasForeignKey("TenantID")
                        .OnDelete(DeleteBehavior.Restrict)
                        .IsRequired()
                        .HasConstraintName("FK_InvoiceReconciliationFile_Tenant_FkTenantID");

                    b.Navigation("Supplier");

                    b.Navigation("Tenant");
                });

And references again here:

            modelBuilder.Entity("CloudManagementTool.DataAccess.Model.Supplier.SupInvoiceReconciliationFileInvoice", b =>
                {
                    b.HasOne("CloudManagementTool.DataAccess.Model.Supplier.SupInvoiceReconciliationFile", "ReconcilationFile")
                        .WithMany("Invoices")
                        .HasForeignKey("ReconcilationFileID")
                        .OnDelete(DeleteBehavior.Restrict)
                        .IsRequired()
                        .HasConstraintName("FK_InvoiceReconciliationFileInvoice_InvoiceReconciliationFile_FkReconcilationFileID");

                    b.HasOne("Neusta.Shared.DataAccess.EntityFramework.TenantFilter.Entities.TenantEntity", "Tenant")
                        .WithMany()
                        .HasForeignKey("TenantID")
                        .OnDelete(DeleteBehavior.Restrict)
                        .IsRequired()
                        .HasConstraintName("FK_InvoiceReconciliationFileInvoice_Tenant_FkTenantID");

                    b.Navigation("ReconcilationFile");

                    b.Navigation("Tenant");
                });

And again somwhere later in the file:

            modelBuilder.Entity("CloudManagementTool.DataAccess.Model.Supplier.SupInvoiceReconciliationFile", b =>
                {
                    b.Navigation("Invoices");
                });

@schuettecarsten schuettecarsten changed the title Exception when creating new migration after updating from 6.0-rc.1 to 6.0-rc.2 Exception when creating new migration after updating from 6.0-rc.1 to 6.0-rc.2 or 6.0-rtm (daily) Oct 21, 2021
@ajcvickers ajcvickers removed this from the 6.0.0 milestone Oct 22, 2021
@ajcvickers ajcvickers removed type-bug closed-fixed The issue has been fixed and is/will be included in the release indicated by the issue milestone. area-migrations labels Oct 22, 2021
@ajcvickers
Copy link
Member

@schuettecarsten It's very hard to tell what is going on with just the snippets of the snapshots. I know it may take some work, but if you can, then please attach a small, runnable project or post a small, runnable code listing that reproduces what you are seeing so that we can investigate.

@schuettecarsten
Copy link
Author

schuettecarsten commented Oct 25, 2021

@ajcvickers I will do my very best, but it might take one or two days before I have time for this.

Meanwhile, do you provide nightly/daily "checked" builds that contain full debug infos? Maybe looking into this with some more details from the callstack helps me to find our what's wrong. If not, I can try to build the packages myself.

When looking into the changes between rc.1 and rc.2, I can only see changes about [Column(Order)]. In commit 637b9a2, line 1067 of MigrationsModelDiffer.cs it looks like an annotation is added to the old column - which will modify the old model - and not to the alterColumnOperation. This annotation is not removed.

@ajcvickers
Copy link
Member

@schuettecarsten What kind of debug information are you looking for? There is very little difference between the release and debug builds. I would not expect using a debug build to be helpful here.

@schuettecarsten
Copy link
Author

@ajcvickers The exact line number of the exception and non-optimized code to show the complete callstack without inlining.

@schuettecarsten
Copy link
Author

@ajcvickers Ok, the rc.2-tag/branch was deleted and the main branch already references 6.0.0 final packages, so I have to wait for these dependencies to be available. I will check this later next week.

@roji
Copy link
Member

roji commented Oct 26, 2021

@schuettecarsten rc2 can be found via the tag v6.0.0-rc.2.21480.5. But the branch release/6.0 contains the very latest state of the 6.0 release, and is very likely to be almost identical to the actual release. You should be able to clone that branch and build it.

@ajcvickers
Copy link
Member

EF Team Triage: Closing this issue as the requested additional details have not been provided and we have been unable to reproduce it.

BTW this is a canned response and may have info or details that do not directly apply to this particular issue. While we'd like to spend the time to uniquely address every incoming issue, we get a lot traffic on the EF projects and that is not practical. To ensure we maximize the time we have to work on fixing bugs, implementing new features, etc. we use canned responses for common triage decisions.

@SmithE65
Copy link

This is not the exact scenario as above but I get the same stack trace. I think I have a repro here: https://github.com/SmithE65/TableRename

Maybe not as minimal as could be.

public class ApplicationDbContext : DbContext
{
    public ApplicationDbContext(DbContextOptions options) : base(options)
    {
    }

    public DbSet<Entity1>? Entity1s { get; set; }
    public DbSet<Entity2>? Entity2s { get; set; }
    public DbSet<Entity3>? Entity3s { get; set; }
}

// ....

public class Entity1
{
    public int Id { get; set; }
    public int Entity2Id { get; set; }
    [ForeignKey(nameof(Entity2Id))]
    public Entity2? Entity2Navigation { get; set; }
    public int Entity3Id { get; set; }
    [ForeignKey(nameof(Entity3Id))]
    public Entity3? Entity3Navigation { get; set; }
}

public class Entity2
{
    public int Id { get; set; }
    [InverseProperty(nameof(Entity1.Entity2Navigation))]
    public ICollection<Entity1> Entity1s { get; set; } = new List<Entity1>();
}

public class Entity3
{
    public int Id { get; set; }
    [InverseProperty(nameof(Entity1.Entity3Navigation))]
    public ICollection<Entity1> Entity1s { get; set; } = new List<Entity1>();
}

I create the three entities as above, create an Initial migration, rename Entity1 class and rename Entity1s DbSet, and attempt to create a second migration. Migration fails with below exception and trace:

System.ArgumentException: An item with the same key has already been added. Key: [0, Property: TableRename.Entity1 (Dictionary<string, object>).Entity3Id (no field, int) Indexer Required FK Index]
   at System.Collections.Generic.TreeSet`1.AddIfNotPresent(T item)
   at Microsoft.EntityFrameworkCore.Migrations.Internal.MigrationsModelDiffer.GetSortedProperties(IEntityType entityType, ITable table)
   at Microsoft.EntityFrameworkCore.Migrations.Internal.MigrationsModelDiffer.GetSortedColumns(ITable table)
   at Microsoft.EntityFrameworkCore.Migrations.Internal.MigrationsModelDiffer.Add(ITable target, DiffContext diffContext)+MoveNext()
   at Microsoft.EntityFrameworkCore.Migrations.Internal.MigrationsModelDiffer.DiffCollection[T](IEnumerable`1 sources, IEnumerable`1 targets, DiffContext diffContext, Func`4 diff, Func`3 add, Func`3 remove, Func`4[] predicates)+MoveNext()
   at System.Linq.Enumerable.ConcatIterator`1.MoveNext()
   at Microsoft.EntityFrameworkCore.Migrations.Internal.MigrationsModelDiffer.Sort(IEnumerable`1 operations, DiffContext diffContext)
   at Microsoft.EntityFrameworkCore.Migrations.Internal.MigrationsModelDiffer.GetDifferences(IRelationalModel source, IRelationalModel target)
   at Microsoft.EntityFrameworkCore.Migrations.Design.MigrationsScaffolder.ScaffoldMigration(String migrationName, String rootNamespace, String subNamespace, String language)
   at Microsoft.EntityFrameworkCore.Design.Internal.MigrationsOperations.AddMigration(String name, String outputDir, String contextType, String namespace)
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.AddMigrationImpl(String name, String outputDir, String contextType, String namespace)
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.AddMigration.<>c__DisplayClass0_0.<.ctor>b__0()
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.OperationBase.<>c__DisplayClass3_0`1.<Execute>b__0()
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.OperationBase.Execute(Action action)
An item with the same key has already been added. Key: [0, Property: TableRename.Entity1 (Dictionary<string, object>).Entity3Id (no field, int) Indexer Required FK Index]

If I rename just the Entity1 class, create an empty migration, rename the DbSet, and then create a migration with the table rename, everything works as expected. I have to change BOTH for the migration to fail.

@ajcvickers
Copy link
Member

Confirmed repro.

@glenkitchen
Copy link

This worked for me. Undelete entities for now. (Will delete after 6.0.2. released)

@warappa
Copy link

warappa commented Dec 29, 2021

In my case I was replacing the term Download with Asset in my whole model, with the term being used in foreign keys, tables, navigation properties etc.

dotnet ef migrations add ... raised the mentioned exception.

Workaround

  1. Rename foreign keys, navigation properties, collections etc as you wish
  2. In the entity type configurations, ensure the entity-table-names containing this name are still using the old name (i.e. "Downloads", "DownloadTranslations") - read: ensure all .ToTable(...) expressions use the old name, and, if they don't exist, add them.
    (I think foreign key names etc. can already use the new name)
  3. Create a new migration with dotnet ef migrations add ... - no exception
  4. Check generated migration code (no unwanted drops etc)
  5. Rename the tables to their new names (i.e. "Assets", "AssetTranslations")
  6. Create a new migration with dotnet ef migrations add ... - no exception
  7. Check generated migration code (no unwanted drops etc)

Word of Caution

Always have a backup or an integration database to test if the migrations are working as expected, even if the tool doesn't throw an exception.

Infos

EF Core version: 6.0.1
EF Core tool version: 6.0.1

@mattstewart1189
Copy link

@warappa thanks for this. I was able to take care of my renames quickly. I had a few a couple weeks ago that had me pulling my hair out.

@systemutvecklare
Copy link

I also got this error when dropping a table and some related tables. A solution that worked for me when removing an entity and getting this error was to

  1. Remove one navigation property (and corresponding foreign key) at a time from the entity and create a migration for each removal.
  2. When no more navigation properties were left in the entity I could remove it and create a final migration without the error message.

In my case I couldn't wait for a new release in february. I spent a whole day with trial and errors before succeeding

@Tragetaschen
Copy link
Contributor

I know it's fixed and scheduled for Release, but not having access to a preview package for the patch release to unblock the scenarios that are affected is… annoying (see #26632 (comment)).
I wish there was a better way than waiting for several weeks :-/

@ajcvickers
Copy link
Member

@Tragetaschen Agreed. However, you can use the 7.0.0 daily build, which has the 6.0.2 patches in it, and not much else at this point in the release cycle.

@MCFHTAGENTS
Copy link

Silly question - where do I find .NET Core 7? as there is a dependency on that (obviously) for the proposed fix - I have an April deadline for some changes and can't wait until February!

@ajcvickers
Copy link
Member

@MCFHTAGENTS

here is a dependency on that (obviously) for the proposed fix

There should not be any dependency on .NET 7. Where are you seeing that?

@MCFHTAGENTS
Copy link

When trying to install the packages:

NU1202: Package Microsoft.AspNetCore.Identity.EntityFrameworkCore 7.0.0-alpha.1.22069.7 is not compatible with net6.0 (.NETCoreApp,Version=v6.0). Package Microsoft.AspNetCore.Identity.EntityFrameworkCore 7.0.0-alpha.1.22069.7 supports: net7.0 (.NETCoreApp,Version=v7.0)

NU1202: Package Microsoft.AspNetCore.DataProtection.EntityFrameworkCore 7.0.0-alpha.1.22069.7 is not compatible with net6.0 (.NETCoreApp,Version=v6.0). Package Microsoft.AspNetCore.DataProtection.EntityFrameworkCore 7.0.0-alpha.1.22069.7 supports: net7.0 (.NETCoreApp,Version=v7.0)

@ajcvickers
Copy link
Member

@MCFHTAGENTS Do you need to update your ASP.NET Core packages as well? The EF Core packages should work with .NET 6, without updating any ASP.NET packages.

@MCFHTAGENTS
Copy link

That is very likely! Where do I get them from - assume I need another nuget feed for example?!

@ajcvickers
Copy link
Member

@MCFHTAGENTS They are in the daily build feed shown in the instructions linked above.

@MCFHTAGENTS
Copy link

sorry - we are going to veer off topic (and I got the daily build feed working to the EntityFramework daily) but I now get
'AspNetCoreRuntime.7.0.x64 7.0.0-alpha-1-22069-8' has a package type 'AzureSiteExtension' that is not supported by project
(assuming that is the correct package) and google returns nothing with a AzureSiteExtension (am running VS 2022) so not sure what I need to add to solve that dependency.....

@ajcvickers
Copy link
Member

@MCFHTAGENTS Looks like the daily builds might not work for ASP.NET Core applications.

@MCFHTAGENTS
Copy link

MCFHTAGENTS commented Jan 20, 2022

hmm - oh to get a nightly for 6.0.2.....
OK - adding the entities back in but removing the foreign keys has allowed me to build the migration script.

@Sebazzz
Copy link

Sebazzz commented Jan 21, 2022

There is no nightly for 6.0.2 and using 7.0.0 was not trivial . I resorted to removing the affected tables (that needed deletion) from the snapshot file, then generate the migration.

@shubhamranjan
Copy link

I also got this error when dropping a table and some related tables. A solution that worked for me when removing an entity and getting this error was to

1. Remove one navigation property (and corresponding foreign key) at a time from the entity and create a migration for each removal.

2. When no more navigation properties were left in the entity I could remove it and create a final migration without the error message.

In my case I couldn't wait for a new release in february. I spent a whole day with trial and errors before succeeding

To anyone who can't wait for the fix. IMO, this is the easiest workaround available.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-migrations closed-fixed The issue has been fixed and is/will be included in the release indicated by the issue milestone. customer-reported regression Servicing-approved type-bug
Projects
None yet
Development

No branches or pull requests