Git Product home page Git Product logo

Comments (16)

kakins avatar kakins commented on July 17, 2024 2

@borisdj Thank you for the response. Actually I found out the problem. The issue was due to case sensitivity.

For instance my public int CostingAllocationDetailAmountId { get; set; } property in the database is actually named CostingAllocationDetailAmountID. Note the capitalized "D". I had to annotate my model like so:

[Column("CostingAllocationDetailAmountID")]
public int CostingAllocationDetailAmountId { get; set; }

However, Entity Framework does not seem to care that the property names are different. Is there any way to configure BulkExtensions to work like EF, and to ignore case sensitivity?

from efcore.bulkextensions.

borisdj avatar borisdj commented on July 17, 2024

Are columns in db table in same order as their Properties in Entity?
This is required when having OwnedType, and it is described in README:
It also maps OwnedTypes, but when that is used it will work only if Properties of Entity are in the same order as DbColumns, because this is implemented with DataTable class.

from efcore.bulkextensions.

borisdj avatar borisdj commented on July 17, 2024

I have done some more test and have reproduced the issue even with correct order of Properties.
So it's a bug regarding NullableTypes that you have correctly found part of fix.
The same fix done in line 175 has to be done also in line 205.
I will soon publish new NuGet version with this fix.

from efcore.bulkextensions.

borisdj avatar borisdj commented on July 17, 2024

NuGet 2.1.4 published.

from efcore.bulkextensions.

tjackadams avatar tjackadams commented on July 17, 2024

@borisdj Thanks for that! is it possible we can reopen this issue?

I've updated the properties in the class to be the same order as what is in the database, however i'm still getting this exception.

I've placed a breakpoint in the GetDataTable<T> method and manually iterated the properties foreach loop and they seem to match up with the order that is in the db table. Is there anyway to dump the Column Mappings that you know of or is there anyway i can debug this further?

Also, i'm assuming the Include/Exclude properties options of the BulkConfig are ignored when using OwnedTypes?

from efcore.bulkextensions.

borisdj avatar borisdj commented on July 17, 2024

Did you try with 2.1.4 Nuget?
If you still have error with that version than you can reopen the issue and add code of entity class where it happens.
Regarding Include/Exclude properties they should work even with OwnedTypes.

from efcore.bulkextensions.

tjackadams avatar tjackadams commented on July 17, 2024

Yeah thats with the latest NuGet package.
The exception NotSupportedException: DataSet does not support System.Nullable<>. is gone, but the original The given ColumnMapping does not match up with any column in the source or destination exception still remains.
I don't have the ability to reopen issues so shall i just post a new one with the class details?

from efcore.bulkextensions.

borisdj avatar borisdj commented on July 17, 2024

Issue Reopened

from efcore.bulkextensions.

tjackadams avatar tjackadams commented on July 17, 2024

Ok, so here are the classes. It's quite a deep graph however the only classes i am interested in Adding/Updating are the CentraStageDevice, CentraStageDeviceType, CentraStageDeviceUdf. The other collection properties i.e CentraStageDeviceLogicalDisk and Added/Updated separately.

Base Class

    [JsonConverter(typeof(CentraStageDeviceBaseConverter))]
    public abstract class CentraStageDeviceBase : Entity<long>
    {
        public bool A64Bit { get; set; }
        public string CagVersion { get; set; }

        [JsonConverter(typeof(EpochConverter))]
        public DateTime CreationDate { get; set; }

        public bool Deleted { get; set; }
        public string Description { get; set; }
        public string DeviceClass { get; set; }

        [JsonIgnore]
        public CentraStageDeviceType DeviceType { get; set; } = new CentraStageDeviceType();

        [JsonIgnore]
        public string Category { get;set; }

        [JsonIgnore]
        public string Type {get; set; }

        public string DisplayVersion { get; set; }
        public string Domain { get; set; }
        public string ExtIpAddress { get; set; }
        public string Hostname { get; set; }
        public string IntIpAddress { get; set; }

        [JsonConverter(typeof(EpochConverter))]
        public DateTime? LastAuditDate { get; set; }

        public string LastLoggedInUser { get; set; }

        [JsonConverter(typeof(EpochConverter))]
        public DateTime? LastReboot { get; set; }

        [JsonConverter(typeof(EpochConverter))]
        public DateTime? LastSeen { get; set; }

        public bool Online { get; set; }
        public string OperatingSystem { get; set; }
        public string PortalUrl { get; set; }
        public bool RebootRequired { get; set; }
        public virtual CentraStageSite Site { get; set; }
        public int SiteId { get; set; }
        public string SiteName { get; set; }
        public string SiteUid { get; set; }
        public bool SnmpEnabled { get; set; }
        public bool Suspended { get; set; }
        public CentraStageDeviceUdf Udf { get; set; }
        public string Uid { get; set; }
    }

Derived Class

    [JsonConverter(typeof(CentraStageDeviceBaseConverter))]
    public class CentraStageDevice : CentraStageDeviceBase
    {
        public virtual ICollection<CentraStageDeviceLogicalDisk> LogicalDisks { get; set; }
        public virtual ICollection<CentraStageDeviceSoftware> Software { get; set; }
        public virtual CentraStageDeviceSystemInfo SystemInfo { get; set; }

        // some methods
    }

Column Order

SELECT [Id]
      ,[A64Bit]
      ,[CagVersion]
      ,[CreationDate]
      ,[Deleted]
      ,[Description]
      ,[DeviceClass]
      ,[DisplayVersion]
      ,[Domain]
      ,[ExtIpAddress]
      ,[Hostname]
      ,[IntIpAddress]
      ,[LastAuditDate]
      ,[LastLoggedInUser]
      ,[LastReboot]
      ,[LastSeen]
      ,[Online]
      ,[OperatingSystem]
      ,[PortalUrl]
      ,[RebootRequired]
      ,[SiteId]
      ,[SiteName]
      ,[SiteUid]
      ,[SnmpEnabled]
      ,[Suspended]
      ,[Uid]
      ,[Category]
      ,[Type]
      ,[Udf_Udf1]
      ,[Udf_Udf10]
      ,[Udf_Udf2]
      ,[Udf_Udf3]
      ,[Udf_Udf4]
      ,[Udf_Udf5]
      ,[Udf_Udf6]
      ,[Udf_Udf7]
      ,[Udf_Udf8]
      ,[Udf_Udf9]
  FROM [PortalCore].[dbo].[CentraStageDevices]

EF Core Configuration

    public class CentraStageDeviceBaseConfiguration : IEntityTypeConfiguration<CentraStageDeviceBase>
    {
        public void Configure(EntityTypeBuilder<CentraStageDeviceBase> builder)
        {
            builder
                .Ignore(p => p.DeviceType);

            builder
                .OwnsOne(p => p.Udf);

            builder
                .ToTable("CentraStageDevices")
                .HasDiscriminator<string>("DeviceClass")
                .HasValue<CentraStageDevice>("device")
                .HasValue<CentraStageEsxiHost>("esxihost")
                .HasValue<CentraStagePrinter>("printer");
        }
    }
    public class CentraStageDeviceConfiguration : IEntityTypeConfiguration<CentraStageDevice>
    {
        public void Configure(EntityTypeBuilder<CentraStageDevice> builder)
        {
            builder
                .Property(p => p.Id)
                .ValueGeneratedNever();

            builder
                .HasBaseType<CentraStageDeviceBase>();

            builder
                .HasMany(p => p.Software)
                .WithOne(c => c.Device)
                .HasForeignKey(c => c.DeviceId);

            builder
                .HasMany(p => p.LogicalDisks)
                .WithOne(c => c.Device)
                .HasForeignKey(c => c.DeviceId);

            builder
                .HasOne(p => p.SystemInfo)
                .WithOne(c => c.Device)
                .HasForeignKey<CentraStageDeviceSystemInfo>(c => c.DeviceId);

            builder
                .HasIndex(p => p.Uid);
        }
    }

As you can probably tell from the class definition, there are a lot more classes that i haven't included just for brevity, but if you want them then just let me know.

So essentially when the BulkInsertOrUpdateAsync method is called, it is receiving a collection of CentraStageDevice.

I also tried using the PropertiesToInclude option in the bulk config and just including the Id column, but this failed with the same error.

from efcore.bulkextensions.

borisdj avatar borisdj commented on July 17, 2024

You have here pretty complex composition.
Anyway I have added this into Test of the library and simplified it a little bit by reducing number of properties but still kept all special elements and structure. Some configuration is done with Annotations.
Then I have found 2 problems and a way to fix them.

First issue was when 'CentraStageDevice' is derived from 'CentraStageDeviceBase' on creating database columns of that table were not in proper order.
This is some EFCore issue.
So I have temporary commented CentraStageDevice and renamed CentraStageDeviceBase to CentraStageDevice, effectively eliminating inheritance. Then I have created DB and after that restored inheritance structure.

Second issue is with inheritance of :Entity<long> where PK Id when in Parent class Entity will be at end of list of properties and in Db Column is first one so there will be order mismatch.
Therefore I have removed that inheritance and moved PK directly in class:
public long Id { get; set; }

After that BulkInsert worked.

This is the code:

public class TestContext : DbContext
{
    // ...
    
    public DbSet<CentraStageDevice> CentraStageDevices { get; set; }
    public DbSet<CentraStageDeviceLogicalDisk> CentraStageDeviceLogicalDisks { get; set; }
    public DbSet<CentraStageDeviceSystemInfo> CentraStageDeviceSystemInfos { get; set; }
    public DbSet<CentraStageDeviceType> CentraStageDeviceTypes { get; set; }
    public DbSet<CentraStageSite> CentraStageSites { get; set; }
    
    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        // ...
        
        modelBuilder.Entity<CentraStageDevice>()
            .ToTable("CentraStageDevice")
            .HasDiscriminator<string>("DeviceClass")
            .HasValue<CentraStageDevice>("device");

        modelBuilder.Entity<CentraStageDevice>().HasMany(p => p.LogicalDisks)
            .WithOne(c => c.Device)
            .HasForeignKey(c => c.DeviceId);

        modelBuilder.Entity<CentraStageDevice>()
            .HasOne(p => p.SystemInfo)
            .WithOne(c => c.Device)
            .HasForeignKey<CentraStageDeviceSystemInfo>(c => c.DeviceId);

        modelBuilder.Entity<CentraStageDevice>()
            .HasIndex(p => p.Uid);
    }
}

public class CentraStageDeviceLogicalDisk
{
    public int Id { get; set; }
    public string LogicalDisk { get; set; }

    public CentraStageDevice Device { get; set; }
    public long DeviceId { get; set; }
}

public class CentraStageDeviceSystemInfo
{
    public int Id { get; set; }
    public string SystemInfo { get; set; }

    public CentraStageDevice Device { get; set; }
    public long DeviceId { get; set; }
}

public class CentraStageDeviceType
{
    public int Id { get; set; }
    public string DeviceType { get; set; }
}

public class CentraStageSite
{
    public int Id { get; set; }
    public string Site { get; set; }
}

[Owned]
public class CentraStageDeviceUdf
{
    public string Udf1 { get; set; }
    public string Udf2 { get; set; }
}

public class CentraStageDevice : CentraStageDeviceBase
{
    public virtual ICollection<CentraStageDeviceLogicalDisk> LogicalDisks { get; set; }
    public virtual CentraStageDeviceSystemInfo SystemInfo { get; set; }
}

/*public abstract class Entity<T>
{
    [DatabaseGenerated(DatabaseGeneratedOption.None)]
    public T Id { get; set; }
}*/

public abstract class CentraStageDeviceBase// : Entity<long>
//public class CentraStageDevice
{
    [DatabaseGenerated(DatabaseGeneratedOption.None)]
    public long Id { get; set; }

    public bool A64Bit { get; set; }
    public string CagVersion { get; set; }

    public string DeviceClass { get; set; }

    [NotMapped]
    public CentraStageDeviceType DeviceType { get; set; } = new CentraStageDeviceType();

    public virtual CentraStageSite Site { get; set; }
    public int? SiteId { get; set; }

    public CentraStageDeviceUdf Udf { get; set; }

    public string Uid { get; set; }
    
    // Added here temporary for DB creation
    //public virtual ICollection<CentraStageDeviceLogicalDisk> LogicalDisks { get; set; }
    //public virtual CentraStageDeviceSystemInfo SystemInfo { get; set; }
}
public void OperationsTest(bool isBulkOperation)
{
    //...
    using (var context = new TestContext(ContextUtil.GetOptions()))
    {
        var entities = new List<CentraStageDevice>();
        for (int i = 1; i < EntitiesNumber; i++)
        {
            var entity = new CentraStageDevice
            {
                Id = i,
                A64Bit = true,
                DeviceClass = "DC1"
                Udf = new CentraStageDeviceUdf(),

            };
            entities.Add(entity);
        }
        context.BulkInsert(entities);
    }
}

from efcore.bulkextensions.

kakins avatar kakins commented on July 17, 2024

I'm having this same issue, only my model is not very complex. I'm simply trying to use BulkInsert as described in the documentation.

Is the navigation property an issue?

image

    public class CostingAllocationDetailAmount
    {
        [Key]
        public int CostingAllocationDetailAmountId { get; set; }
        public int CostingAllocationDetailId { get; set; }
        public DateTime DistributedWorkPeriodStartDate { get; set; }
        public DateTime DistributedWorkPeriodEndDate { get; set; }
        public decimal Pay { get; set; }
        public decimal Longevity { get; set; }
        public decimal BenefitsEarningsWithLongevity { get; set; }
        public decimal Retirement { get; set; }
        public decimal PreTaxEarnings { get; set; }
        public decimal GIPAmount { get; set; }
        public decimal OASIAmount { get; set; }
        public decimal OAHIAmount { get; set; }
        public decimal WCIAmount { get; set; }
        public decimal UCIAmount { get; set; }
        public decimal ACAPTaxAmount { get; set; }
        public decimal MeritAmount { get; set; }
        public decimal PromotionAmount { get; set; }
        public decimal AdditionalAdjustmentAmount { get; set; }
        public decimal CurrentFteMonthlyRate { get; set; }
        public decimal CurrentFteSourceAmount { get; set; }
        public decimal AnnualWorkPeriodMultiplier { get; set; }
        public decimal TotalRateAdjustmentAmount { get; set; }
        public decimal IntervalPercentOfAnnualWorkPeriod { get; set; }

        [ForeignKey(nameof(CostingAllocationDetailId))]
        public virtual CostingAllocationDetail Detail { get; set; }
    }
}
``

from efcore.bulkextensions.

borisdj avatar borisdj commented on July 17, 2024

I have added you model in Test (just reduced number of decimal Properties) and ti works fine.

public class TestContext : DbContext
{
    public DbSet<CostingAllocationDetailAmount> CostingAllocationDetailAmounts { get; set; }
    public DbSet<CostingAllocationDetail> CostingAllocationDetails { get; set; }
}

public class CostingAllocationDetailAmount
{
    [Key]
    public int CostingAllocationDetailAmountId { get; set; }

    public int CostingAllocationDetailId { get; set; }

    public DateTime DistributedWorkPeriodStartDate { get; set; }
    public DateTime DistributedWorkPeriodEndDate { get; set; }

    public decimal Pay { get; set; }
    public decimal Longevity { get; set; }

    [ForeignKey(nameof(CostingAllocationDetailId))]
    public virtual CostingAllocationDetail Detail { get; set; }
}

public class CostingAllocationDetail
{
    public int CostingAllocationDetailId { get; set; }
    public string Name { get; set; }
}

private void RunInsert(bool isBulkOperation)
{
    using (var context = new TestContext(ContextUtil.GetOptions()))
    {
        var entities = new List<CostingAllocationDetailAmount>();
        for (int i = 1; i < EntitiesNumber; i++)
        {
            var entity = new CostingAllocationDetailAmount
            {
                CostingAllocationDetailId = 1,
                DistributedWorkPeriodStartDate = DateTime.Now,
                DistributedWorkPeriodEndDate = DateTime.Now,
                Pay = 10.0m,
                Longevity = 20.0m,
            };
            entities.Add(entity);
        }

        using (var transaction = context.Database.BeginTransaction())
        {
            context.BulkInsert(entities);
            transaction.Commit();
        }
    }
}

Please reproduce it in Test and write entire code of it.

from efcore.bulkextensions.

MaceWindu avatar MaceWindu commented on July 17, 2024

This is underlying SqlBulkCopy limitation, not BulkExtensions's, so you need to specify column names in your mapping properly if you want to use them with bulk copy

from efcore.bulkextensions.

kakins avatar kakins commented on July 17, 2024

But it is probably best to clarify that it is not "improper", according to EF, to specify property names with different casing than your SQL table columns.

In addition, it is unknown to the user what the underlying technology the extension is using -- he/she only knows that it is supposed to work with EF and most likely expects it to work seamlessly with standard EF conventions.

That being the case, is this limitation documented anywhere? I'd suggest that it at least be prominent in the "getting started" sections of the docs.

from efcore.bulkextensions.

MaceWindu avatar MaceWindu commented on July 17, 2024

EF works with SQL and MSSQL parses it using master database collation, which is usually case-insensitive. If you will configure your server to use case-sensitive collation on master - it will also break for EF if names in your mappings doesn't match database names.

BulkCopy is a bit different mechanism and it is implemented to be case-sensitive and nothing could be changed here

from efcore.bulkextensions.

kakins avatar kakins commented on July 17, 2024

Ah, that actually makes a lot of sense now. Thanks for the clarification!

from efcore.bulkextensions.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.