Sinequa For Azure (S4A) is a set of capabilities and dedicated features designed for Azure.

This repository contains resources for deploying a S4A grid. Sinequa For Azure features are included in the Sinequa distribution 11.7.0 version and above.

In case you don't want to install Sinequa For Azure using the PowerShell and Terraform scripts, you can read this guide: S4A - Deploy using agnostic Infrastructure as Code

0. Prerequisistes
   
1. Repository Content
   1.1. Build your own Sinequa image
   1.2. Deploy a Sinequa grid
   
2. Sinequa Azure Features
   2.1. Cloud Init
   2.2. Leverage Storage Account
   2.3. Leverage a Secondary Storage Account
   2.4. Leverage Scale Set for Elasticity
   2.5. Back Up and Restore the Application
   

• Access to https://portal.azure.com
• Install Azure CLI: https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-windows?tabs=azure-cli
• Install Azure Powershell: https://docs.microsoft.com/fr-fr/powershell/azure/install-az-ps
• Install Terraform: https://www.terraform.io/downloads.html

This repository contains:

• Packer scripts to build your own Sinequa image
• Terraform samples to deploy a Sinequa grid

The cloud init features are some capabilities used upon VM deployment to initialize an out-of-the-box Sinequa node that is automaticaly registered into a grid with some roles enabled (like engine, indexer, etc.).

You must set the SINEQUA_CLOUD environment variable before starting the Sinequa service to enable all Sinequa For Azure features such as the cloud init.

Cloud tags are Azure tags used on Azure resources. They are used to run some specific init tasks for a particular VM or VMSS.

Note: Cloud tags are available as "Environment Variables" in the product with the _CloudTag_ prefix. E.g The sinequa-node tag value is accessible through the _CloudTag_sinequa-node Environment Variable.

• Cloud variables are Azure blobs stored in the storage account. They are used to declare global variables in the configuration.

Note: Cloud variables are available as "Environment Variables" in the product with the _CloudVar_ prefix. E.g The sinequa-beta cloud var value is accessible through the _CloudVar_sinequa-beta Environment Variable.

• Cloud secrets are secrets stored in the key vault defined in cloud variables. They are used to store sensitive data.

Note: Cloud secrets begining with the sinequa-env-prefix are available as "Environment Variables" in the product with the _CloudSecret_ prefix. E.g The sinequa-env-my-secret secret value is accessible through the _CloudSecret_sinequa-env-my-secret Environment Variable.

• Cloud Aliases are Azure blobs stored in the primary storage account. They are used to declare Sinequa aliases in the configuration.

Cloud Init Scripts are custom PowerShell scripts that are located in the primary storage account. These scripts are executed by the Cloud Init service before the sinequa.service service starts. All scripts must be located in a cloudinit-scripts folder. These folders could be in two levels:

1. Org level - {org name}/cloudinit-scripts
2. Grid level - {org name}/grids/{grid name}/cloudinit-scripts

All scripts are executed in alphabetical order of the filename, and Grid level scripts overwrite Org level scripts with the same name.

To reduce the cost of the disk usage and have a better reliability and availabilty on data, an Azure Blob storage account is broadly used for all data that do not require high I/O performances.

If a sinequa-data-storage-url cloud tag is provided, the components below will automatically synchronize data from disk to Azure Storage account.

It concerns:

• Document cache store
• User settings
• Registry
• Configuration
• Audit store
• Log store

As Blobs have not the same size and not the same frequency of access, a second storage could be used for leveraging cheaper SKUs.

If a sinequa-data-storage-url cloud tag is provided (primary storage), and if this storage contains an org var called sinequa-secondary then some blobs will be moved to this secondary Azure Storage account.

• Primary Storage with the Premium sku for fast access and small content (blob < 10ko)
• Secondary Storage with the Standard Hot sku for large content (sha + queue + logs)

• Primary Storage (Premium --> blob <= 10K)

  • /{org name}
    • /var
    • /cloudinit-scripts
      • /sinequa-secondary => contains url of the secondary storage
    • /grids
      • /{grid name}
        • /var
        • /Document cache store
        • /User settings
        • /Registry
        • /Configuration
      • /cloudinit-scripts

• Secondary Storage (Standard Hot)

  • /{org name}
    • /grids
      • /{grid name}
        • /audit
        • /log
        • /queue-volumes
    • /sha
      • /blobs

To reduce the cost of VM usage and control the indexing workload, scale set is used to scale up & down the number of indexers based on the indexing workload.

Thanks to Storage Accounts, you can easily back up and restore a complete grid with consistency between components.

• You can back up the engine directly into an Azure Storage acccount.

• and restore