Git Product home page Git Product logo

joker's Introduction

Data change notifications from SQL Server via SqlTableDependency, OData and Redis to different .NET clients (WinUI3 - UWP and Win32 apps, WPF, Blazor Wasm, etc). Blazor Wasm notifications are redirected with SignalR.

Joker in action

Set docker-compose.csproj as startup project in Joker.sln

Joker Model-View-ViewModel:

Reactive view models for data changes

Install-Package Joker.MVVM

Joker OData:

Plumbing code for OData web services. Support for batching and end points. Please help out the community by sharing your suggestions and code improvements:

Preview:

Redis TableDependency status notifier SQL server data changes refresher via Redis with End to end reconnections

SqlTableDependency.Extensions

The SqlTableDependency.Extensions .NET package is a library that provides convenient and efficient ways to monitor and receive real-time notifications for changes in SQL Server database tables. It is built on top of the SqlTableDependency library and extends its functionality.

The main purpose of SqlTableDependency.Extensions is to simplify the process of setting up and handling database table change notifications in .NET applications.

With this package, you can easily subscribe to table changes and receive notifications in your application whenever a row is inserted, updated, or deleted in a specified SQL Server table.

Install:

https://www.nuget.org/packages/SqlTableDependency.Extensions/

Install-Package SqlTableDependency.Extensions

or

dotnet add package SqlTableDependency.Extensions --version 3.0.0

See:

Following package is based on christiandelbianco's SqlTableDependency: https://github.com/christiandelbianco/monitor-table-change-with-sqltabledependency

SqlTableDependency.Extension.SqlTableDependencyProvider provides periodic reconnections in case of any error, like lost connection etc.

Currently there are 3 LifetimeScopes:

UniqueScope:

If the connection is lost, database objects will only be deleted after a timeout period. Upon reconnection, the database objects are recreated, but only if the conversation handle no longer exists.Otherwise, the database objects are preserved and reused. If the application was closed without cleaning the conversation, it will be reused upon app restart. This ensures that data changes within the timeout period are not lost, and messages will be delivered after the reconnection.

ApplicationScope:

In case that the connection is lost, database objects will be deleted only after timeout period. After reconnection the database objects are recreated in case that the conversation handle does not exist anymore. Otherwise the database objects are preserved and reused. If the application was closed the conversation will not continue after app restart. You shouldn't lost data changes within the timeout period. The messages will be delivered after the reconnection.

ConnectionScope:

If the connection is lost, the database objects will be deleted either after a timeout period or during disposal. Upon each reconnection, the database objects are recreated.

Wiki Samples

Docker for external dependencies:

MS SQL Server 2017:

docker run --name sql -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=<YourNewStrong@Passw0rd>" -p 1401:1433 -d mcr.microsoft.com/mssql/server:2017-latest

Redis latest:

docker run --name redis-server -p 6379:6379 -d redis

Examples Entity Framework migrations:

Package Manager Console (Default project => Examples\Samples.Data):

Update-Database -ConnectionString "Server=127.0.0.1,1401;User Id = SA;Password=<YourNewStrong@Passw0rd>;Initial Catalog = Test;" -ConnectionProviderName "System.Data.SqlClient" -ProjectName Sample.Data -verbose

Basic usage:

Enable Service Broker in MS SQL SERVER

ALTER DATABASE [DatabaseName]
    SET ENABLE_BROKER WITH ROLLBACK IMMEDIATE;

// C#

  public class Product
  {
      public int Id { get; set; }
      public string Name { get; set; }
  } 
  
  using SqlTableDependency.Extensions;
  using SqlTableDependency.Extensions.Enums;
  
  internal class ProductsSqlTableDependencyProvider : SqlTableDependencyProvider<Product>
  {
    private readonly ILogger logger;

    internal ProductsSqlTableDependencyProvider(ConnectionStringSettings connectionStringSettings, IScheduler scheduler, ILogger logger)
      : base(connectionStringSettings, scheduler, LifetimeScope.UniqueScope)
    {
      this.logger = logger ?? throw new ArgumentNullException(nameof(logger));
    }

    internal ProductsSqlTableDependencyProvider(string connectionString, IScheduler scheduler, ILogger logger)
      : base(connectionString, scheduler, LifetimeScope.UniqueScope)
    {
      this.logger = logger ?? throw new ArgumentNullException(nameof(logger));
    }

    protected override ModelToTableMapper<Product> OnInitializeMapper(ModelToTableMapper<Product> modelToTableMapper)
    {
      modelToTableMapper.AddMapping(c => c.Id, nameof(Product.Id));

      return modelToTableMapper;
    }

    protected override void OnInserted(Product product)
    {
      base.OnInserted(product);

      logger.Trace("Entity was added");

      LogChangeInfo(product);
    }

    protected override void OnUpdated(Product product, Product oldValues)
    {
      base.OnUpdated(entity, oldValues);

      logger.Trace("Entity was modified");

      LogChangeInfo(product);
    }

    protected override void OnDeleted(Product product)
    {
      base.OnDeleted(product);

      logger.Trace("Entity was deleted");

      LogChangeInfo(product);
    }

    private void LogChangeInfo(Product product)
    {
      Console.WriteLine(Environment.NewLine);

      Console.WriteLine("Id: " + product.Id);
      Console.WriteLine("Name: " + product.Name);

      Console.WriteLine("#####");
      Console.WriteLine(Environment.NewLine);
    }
    
    protected override void OnError(Exception exception)
    {
      logSource.Error(exception);
    }
  }

//Program.cs

using System.Configuration;
using System.Reactive.Concurrency;

namespace SqlTableDependency.Extensions.Sample
{
  class Program
  {
    static void Main(string[] args)
    {
      var connectionString = ConfigurationManager.ConnectionStrings["FargoEntities"].ConnectionString;
      
      using var productsProvider = new ProductsSqlTableDependencyProvider(connectionString, ThreadPoolScheduler.Instance, new ConsoleLogger());
      productsProvider.SubscribeToEntityChanges();
      
      Console.ReadKey();
    }
  }
}

Joker.Redis

SqlServer PubSub notifications via Redis and SqlTableDependencyProvider extension

Install-Package Joker.Redis

Download and run redis-server (https://redis.io/download) or use Docker (see above).

Server side:

public class ProductSqlTableDependencyRedisProvider : SqlTableDependencyRedisProvider<Product>
{
  public ProductSqlTableDependencyRedisProvider(ISqlTableDependencyProvider<Product> sqlTableDependencyProvider, IRedisPublisher redisPublisher) 
    : base(sqlTableDependencyProvider, redisPublisher)
  {
  }
}
string redisUrl = ConfigurationManager.AppSettings["RedisUrl"]; //localhost

var redisPublisher = new RedisPublisher(redisUrl);
await redisPublisher.PublishAsync("messages", "hello");

using var productsProvider = new ProductsSqlTableDependencyProvider(connectionString, ThreadPoolScheduler.Instance, new ConsoleLogger());

using var productSqlTableDependencyRedisProvider = new ProductSqlTableDependencyRedisProvider(productsProvider, redisPublisher, ThreadPoolScheduler.Instance)
  .StartPublishing();

Client side:

private static async Task<RedisSubscriber> CreateRedisSubscriber(string redisUrl)
{
  var redisSubscriber = new RedisSubscriber(redisUrl);

  await redisSubscriber.Subscribe(channelMessage => { Console.WriteLine($"OnNext({channelMessage.Message})"); },
  "messages");

  await redisSubscriber.Subscribe(channelMessage =>
  {
    var recordChange = JsonConvert.DeserializeObject<RecordChangedNotification<Product>>(channelMessage.Message);
    Console.WriteLine($"OnNext Product changed({recordChange.ChangeType})");
    Console.WriteLine($"OnNext Product changed({recordChange.Entity.Id})");
  }, nameof(Product) + "-Changes");

  await redisSubscriber.Subscribe(channelMessage =>
  {
    var tableDependencyStatus = JsonConvert.DeserializeObject<TableDependencyStatus>(channelMessage.Message);
    Console.WriteLine($"OnNext tableDependencyStatus changed({tableDependencyStatus})");
  }, nameof(Product) + "-Status");

  redisSubscriber.WhenIsConnectedChanges.Subscribe(c => Console.WriteLine($"REDIS is connected: {c}"));

  return redisSubscriber;
}

How to put it all together

Try out samples

    private static void CreateReactiveProductsViewModel()
    {
      var reactiveData = new ReactiveData<Product>();
      var redisUrl = ConfigurationManager.AppSettings["RedisUrl"];
      using var entitiesSubscriber = new DomainEntitiesSubscriber<Product>(new RedisSubscriber(redisUrl), reactiveData);

      string connectionString = ConfigurationManager.ConnectionStrings["FargoEntities"].ConnectionString;

      var reactiveProductsViewModel = new ReactiveProductsViewModel(new SampleDbContext(connectionString),
        reactiveData, new WpfSchedulersFactory());

      reactiveProductsViewModel.SubscribeToDataChanges();

      reactiveProductsViewModel.Dispose();
    }

"Buy Me A Coffee"

joker's People

Contributors

dependabot[bot] avatar tomasfabian avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

joker's Issues

Upgrading persistent queries

On the Kafka.DotNet.ksqlDB.KSql.Linq package, are there any plans to support CREATE OR REPLACE STREAM statements when using CreateQueryStream<T>()?
I tried to find a way to pass some kind of parameter but couldn't find one.
Thanks!

Error logging issue

Hi

I had a situation recently where the SqlTableDependencyOnError method was executed but the OnError method which takes in an exception object was never executed when the application abruptly stopped.

This OnError method is important as I have an event source which I log into the windows event viewer for SCOM updates.

Thanks,
Rajath

Maintain conversation

Not an issue, per se.
We are using SqlTableDependency have an issue with the drop and recreate of triggers and broker items each time it "hiccups" A windows service makes use of the project to listen to the broker. If our service idels or drops and has to be restarted, the drop and create from SqlTableDependecny fires...during that time, the db is nearly unusable. Your extension mentions the ability to maintain that conversation handle. Would your extension help us in those situations?

[Joker.OData] - UseAuthorization throws InvalidOperationException (SignalR hub)

UseAuthorization should be called between UseRouting and UseEndpoints:

An unhandled exception has occurred while executing the request.
System.InvalidOperationException: Endpoint /dataChangesHub/negotiate contains authorization metadata, but a middleware was not found that supports authorization.
Configure your application startup by adding app.UseAuthorization() inside the call to Configure(..) in the application startup code. The call to app.UseAuthorization() must appear between app.UseRouting() and app.UseEndpoints(...).
at Microsoft.AspNetCore.Routing.EndpointMiddleware.ThrowMissingAuthMiddlewareException(Endpoint endpoint)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware.Invoke(HttpContext httpContext)
at Joker.OData.Middleware.Logging.ErrorLoggerMiddleware.Invoke(HttpContext context) in C:\Github\SqlTableDependency.Extensions\Joker.OData\Middleware\Logging\ErrorLoggerMiddleware.cs:line 23
at Microsoft.AspNet.OData.Batch.ODataBatchMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)

Getting old values with an update event

In that moment it is not possible to get the former data row state during an OnUpdate event, even if the IncludeOldValues settings attribute is set true. Is it possible to fix that?

FarServiceUniqueName is not being considered

I have a .NET application, with SignalR integration.
I am using SqlTableDependency.Extensions and using the UniqueScope for the Lifetime. However I want to change the name of the objects on my database (trigger, service, contracts, SP, ...).

protected override SqlTableDependencySettings<Notification> OnCreateSettings()
{
    var settings = base.OnCreateSettings();
    settings.FarServiceUniqueName = "ProjectName";
    settings.IncludeOldValues = true;

    return settings;
}

The code above doesn't appear to change the objects name on the database, and the Environment.MachineName is what is being considered. Should I add anything else to the settings object in order for the database objects to have the FarServiceUniqueName?

Just raise event with the table PK - with no other data.

Hi There,

I'm following your examples and have tried looking though the source code for some answers, but I don't seem to be able to receive an event with just the ID of the target Table.

If I comment out the EXCEPT statement in the trigger created I can get my desired results.

image

Is there a way to do this programmatically?

My on create settings are borrowed from the console example
image

Possible issue with Dispose in SqlTableDependencyWithReconnection

Hello,

We are trying to use your extension to solve the issue with bad triggers / deadlock with original SqlTableDependency component.
IsNemoEqualTrue/monitor-table-change-with-sqltabledependency#188

And after preliminary testing we see it's working just great.
The only issue we have is an exception that is thrown in Dispose method that is currently implemented like:

protected override void Dispose(bool disposing)
{
    if (LifetimeScope != LifetimeScope.UniqueScope)
      DropDatabaseObjects();

    base.Dispose(disposing);
}

for main Dispose call it works fine, but when a container disposes objects, the following is called:

~TableDependency() => this.Dispose(false);

and we get "Handle is not initialized" from System.Data.ProviderBase.DbConnectionInternal.PostPop exception, most probably because parent object is already disposed.

So I think the proper implementation of that Dispose should include disposing check:

protected override void Dispose(bool disposing)
{
    if (disposing && LifetimeScope != LifetimeScope.UniqueScope)
      DropDatabaseObjects();

    base.Dispose(disposing);
}

Could you please check the issue?

Thank you!

Read the DB-Attributes out of the TableAttribute

Hi Tomas,

I use Extensions 2.3 and it works fine if I override the attribute TableName within the SqlTableDependencyProvider<> derivate with the real table name respectively the settings.SchemaName with the real schema name, because my Entity-class is named different to the table name because of different convention (camel case vs. big case, English vs. German).

I used before the original SqlTableDependency. There it was possibe to define the table and schema and column namnes as attributes in the entity class. This brings all definition at one location together. Could you implement it for the Extensions, too?

Thx,
Tobi

FarServiceUniqueName seems to be ignored

I have derived SqlTableDependencyProvider<T> with Unique Scope and overrided OnCreateSettings() as follow :

protected override SqlTableDependencySettings<TEntity> OnCreateSettings()
        {
            var settings = base.OnCreateSettings();
            using var serviceScope = _scopeFactory.CreateScope();

            var findEntityType = serviceScope.ServiceProvider.GetRequiredService<TContext>()
                .Model.FindEntityType(typeof(TEntity));

            var dmlTriggerType = ConfigurationProfiles.Select(e => e.DmlTriggerType)
                .Aggregate((t1, t2) => t1 | t2);
            settings.IncludeOldValues     = true;
            settings.NotifyOn             = dmlTriggerType;
            settings.FarServiceUniqueName = serviceScope.ServiceProvider.GetRequiredService<IConfiguration>().GetSection("FarServiceUniqueName").Value;
            settings.SchemaName           = findEntityType.GetSchema();
            settings.TableName            = findEntityType.GetTableName();
            settings.TimeOut = (int)TimeSpan.FromDays(5).TotalSeconds;
            settings.WatchDogTimeOut = (int)TimeSpan.FromDays(6).TotalSeconds;
            return settings;
        }

But when I start monitoring with SubscribeToEntityChanges();

It seems that SqlTableDependency is instancied as follow :

case LifetimeScope.UniqueScope:
          SqlTableDependencyWithUniqueScope<TEntity> sqlTableDependency = new SqlTableDependencyWithUniqueScope<TEntity>(this.connectionString, this.TableName, this.SchemaName, modelToTableMapper, settings.UpdateOf, settings.Filter, settings.NotifyOn, settings.ExecuteUserPermissionCheck, settings.IncludeOldValues);
          sqlTableDependency.Settings = this.Settings;
          return (ITableDependency<TEntity>) sqlTableDependency;

But the base constructor at the first instruction doesn't know FarServiceUniqueName at the time leading base constructor to store an outdated value wich is not refreshed even if Settings are set afterward

    protected TableDependency(
      string connectionString,
      string tableName = null,
      string schemaName = null,
      IModelToTableMapper<T> mapper = null,
      IUpdateOfModel<T> updateOf = null,
      ITableDependencyFilter filter = null,
      DmlTriggerType dmlTriggerType = DmlTriggerType.All,
      bool executeUserPermissionCheck = true)
    {
  ...
      this._dataBaseObjectsNamingConvention = this.GetBaseObjectsNamingConvention();
  ...
    }

Setting should be passed to SqlTableDependencyWithReconnection constructor with an optional aditionnal parameter added and _dataBaseObjectsNamingConvention should be reset

public SqlTableDependencyWithReconnection(
      string connectionString,
      string tableName = null,
      string schemaName = null,
      IModelToTableMapper<TEntity> mapper = null,
      IUpdateOfModel<TEntity> updateOf = null,
      ITableDependencyFilter filter = null,
      DmlTriggerType notifyOn = DmlTriggerType.All,
      bool executeUserPermissionCheck = true,
      bool includeOldValues = false,
      SqlTableDependencySettings<TEntity> settings)
      : base(connectionString, tableName, schemaName, mapper, updateOf, filter, notifyOn, executeUserPermissionCheck, includeOldValues)
    {
      Settings                         = settings;
      _dataBaseObjectsNamingConvention = GetBaseObjectsNamingConvention();
    }

Or reset the value properly before Start call

Better fix for #188 (not dropping trigger)

Hello Tomas,

In your "temp fix" to IsNemoEqualTrue/monitor-table-change-with-sqltabledependency#188
I see the only difference was to disable dropping all objects when there are no conversation handlers anymore, as that was causing a deadlock, e.g. you removed the below clauses:

DECLARE @conversationHandlerExists INT
SELECT @conversationHandlerExists = COUNT(*) FROM sys.conversation_endpoints WHERE conversation_handle = '{19}';
IF @conversationHandlerExists = 0
BEGIN
    {20}
    RETURN
END

the fix works great, but for cases when there is a transaction that goes beyond watch dog timeout, users still receive a blocking exception "The conversion handle XXX is not found", like here IsNemoEqualTrue/monitor-table-change-with-sqltabledependency#214

What do you think about returning back the clauses above with a single change:

DECLARE @conversationHandlerExists INT
SELECT @conversationHandlerExists = COUNT(*) FROM sys.conversation_endpoints WHERE conversation_handle = '{19}';
IF @conversationHandlerExists = 0
BEGIN
    -- doing nothing, just quit
    RETURN
END

e.g. for cases when we have some dropped conversation handlers but triggers are still staying in DB for some reason (or they appear to be present in DB in the scope of that running long-term transaction), we just silently skip any logic for that trigger.
As a result, users are not blocked from doing updates to the table?

Thanks!

Not Throwing an Exception when StringConnection is wrong or the Broker is not enabled

Hello,

I'm the same person that reported before on the topic:
SqlTableDependency Recover Connection (Reconnecting)

First, I'm happy that I was the origin of your contribution, I want to thank you for this nice work.

I just have tested it right now, and I want to inform you that using the basic usage, if something is wrong on StringConnection or the Service Broker is not enabled, we didn't get an exception, it just not give up an notification.

Thank you again for this nice work.

There is no way to mention the schema name.

Great work on this extension, keep it going.

With the sqltabledependency constructor I could pass the schema name of the table as a parameter. This seems to not be possible with the sqltabledependencyprovider abstract class.

I may be wrong, please can you suggest a way to do this?

SqlException on application restart

I am using SqlTableDependency.Extensions on a .NET Framework 4.8 web MVC application.
I am using a SqlTableDependencyProvider with a lifetime UniqueScope (with a fixed FarServiceUniqueName).

On application restart, the new SqlTableDependencyProvider causes an SQL Exception: "There is already an object named 'dbo_MyTable_MyProjet/StartMessage/Insert' in the database. This causes a crash on my app, since I am doing these dependencies on Application_Start().

Shouldn't the Provider check for these objects in the database, and reuse them?
("If the application was closed without cleaning the conversation, it will be reused upon app restart")

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.