Question: How to: Ruby Tools and Frameworks

Answer:
[RubyMine](https://www.jetbrains.com/ruby/) is a professional IDE developed by Jet Brains that provides support for Ruby, Ruby on Rails and web development.

[Rails](https://rubyonrails.org/) is a web-application framework that includes everything needed to create database-backed web applications according to the [Model-View-Controller (MVC)](https://en.wikipedia.org/wiki/Model-view-controller) pattern. Understanding the MVC pattern is key to understanding Rails. MVC divides your application into three layers: Model, View, and Controller, each with a specific responsibility.

[rbenv](https://github.com/rbenv/rbenv) allows to pick a Ruby version for your application and guarantee that your development environment matches production. Put rbenv to work with Bundler for painless Ruby upgrades and bulletproof deployments.

[Prettier for Ruby](https://prettier.io/) is a plugin for the Ruby programming language and its ecosystem. prettier is an opinionated code formatter that supports multiple languages and integrates with most editors. The idea is to eliminate discussions of style in code review and allow developers to get back to thinking about code design instead.

[Active Admin](https://activeadmin.info/) is a Ruby on Rails framework for creating elegant backends for website administration.

[Capistrano](https://github.com/capistrano/capistrano) is a framework for building automated deployment scripts. Although Capistrano itself is written in Ruby, it can easily be used to deploy projects of any language or framework, be it Rails, Java, or PHP.

[Spree](https://spreecommerce.org/) is an open source E-commerce platform for Rails 6 with a modern UX, optional PWA frontend, REST API, GraphQL, several official extensions and 3rd party integrations.

[Sidekiq](https://sidekiq.org/) is a simple, efficient background processing for Ruby. It uses hreads to handle many jobs at the same time in the same process. It does not require Rails but will integrate tightly with Rails to make background processing dead simple.

[Kaminari](https://github.com/amatsuda/kaminari/wiki) is a  Scope and Engine based, clean, powerful, and customizable  paginator for modern web app frameworks and ORMs.

[React-Rails](https://github.com/reactjs/react-rails) is a flexible tool to use [React](http://facebook.github.io/react/) with Rails. By integrating React.js with Rails views and controllers, the asset pipeline, or webpacker.

[Pry](https://github.com/pry/pry) is a runtime developer console and IRB alternative with powerful introspection capabilities.

[Brakeman](https://brakemanscanner.org/) is a static analysis tool which checks Ruby on Rails applications for security vulnerabilities.

[dotenv](https://github.com/bkeepers/dotenv) is a Ruby gem to load environment variables from `.env`.

[Scientist](https://github.com/github/scientist) is a Ruby library for carefully refactoring critical paths.

[fastlane](https://fastlane.tools/) is a tool written in Ruby for iOS and Android developers to automate tedious tasks like generating screenshots, dealing with provisioning profiles, and releasing your application.

[Fluentd](https://www.fluentd.org/) collects events from various data sources and writes them to files, RDBMS, NoSQL, IaaS, SaaS, Hadoop and so on all written in Ruby.


Question: How to: Flutter Development

Answer:
[Back to the Top](https://github.com/mikeroyal/Windows-11-Guide/blob/main/README.md#table-of-contents)
[Flutter](https://flutter.dev/) is Google's UI toolkit for crafting beautiful, natively compiled applications for mobile(Andorid and iOS), web, and desktop(Windows, MacOS, Linux, and Google Fuchsia) from a single codebase. Flutter works with existing code, is used by developers and organizations around the world, and is free and open source.


Question: How to: Flutter Learning Resources

Answer:
[Flutter Gems](https://fluttergems.dev) is a curated package guide for Flutter which functionally categorizes some of the most useful and popular flutter packages available on pub.dev Flutter Gems A Flutter package landscape guide comprising 1500+ neatly categorized useful and popular packages.

[Dart](https://dart.dev/) is an open-source, scalable programming language, with robust libraries and runtimes, for building web, server, and mobile apps using the Flutter framework.

[Flutter documentation](https://flutter.dev/docs)

[Style Guide for Flutter](https://github.com/flutter/flutter/wiki/Style-guide-for-Flutter-repo)

[Creating your first Flutter app](https://flutter.dev/docs/get-started/codelab)

[Build and release an Android app using Flutter](https://flutter.dev/docs/deployment/android)

[Flutter Tools & techniques](https://flutter.dev/docs/development/tools)

[Dart and Flutter: The Complete Developer's Guide on Udemy](https://www.udemy.com/course/dart-and-flutter-the-complete-developers-guide/)

[Creating an Interactive Story with Flutter on Coursera](https://www.coursera.org/projects/story-creating-flutter)

[Flutter for Beginners course on Pluralsight](https://www.pluralsight.com/courses/flutter-getting-started)

[Flutter Online Training Courses on LinkedIn Learning](https://www.linkedin.com/learning/topics/flutter)

[The Complete Flutter App Development Bootcamp with Dart by App Brewery](https://www.appbrewery.co/p/flutter-development-bootcamp-with-dart)

[Adding Firebase to your Flutter app](https://firebase.google.com/docs/flutter/setup)

[Using Firebase and Firestore with Flutter](https://flutter.dev/docs/development/data-and-backend/firebase)

[Fuchsia Project](https://fuchsia.dev/)

[Getting Started with Fuchsia](https://fuchsia.dev/fuchsia-src/get-started)

[Fuchsia Reference](https://fuchsia.dev/reference)

[Contributing to Fuchsia](https://fuchsia.dev/fuchsia-src/CONTRIBUTING)


Question: How to: Flutter Tools

Answer:
[Firebase](https://firebase.google.com/) is a Backend-as-a-Service (BaaS) app development platform that provides hosted backend services such as a realtime database, cloud storage, authentication, crash reporting, machine learning, remote configuration, and hosting for your static files.

[FlutterFire](https://firebase.flutter.dev/) is a set of [Flutter plugins](https://flutter.io/platform-plugins/) that enable Flutter apps to use [Firebase](https://firebase.google.com/) services. You can follow an example that shows how to use these plugins in the [Firebase for Flutter](https://codelabs.developers.google.com/codelabs/flutter-firebase/index.html#0) codelab.

[FlutterBoost](https://github.com/alibaba/flutter_boost) is a Flutter plugin which enables hybrid integration of Flutter for your existing native apps with minimum efforts.

[Go-flutter](https://github.com/go-flutter-desktop/go-flutter) is a package that brings Flutter to the desktop. project implements the [Flutter's Embedding API](https://github.com/flutter/flutter/wiki/Custom-Flutter-Engine-Embedders) using a single code base that runs on Windows, macOS, and Linux. For rendering, [GLFW](https://github.com/go-gl/glfw) fits the job because it provides the right abstractions over the OpenGL's Buffer/Mouse/Keyboard for each platform.

[Appwrite](https://appwrite.io/) is a secure end-to-end backend server for Web, Mobile, and Flutter developers that is packaged as a set of Docker containers for easy deployment.

[Fluro](https://github.com/theyakka/fluro) is a Flutter routing library that adds flexible routing options like wildcards, named parameters and clear route definitions.


Question: How to: Node.js Development

Answer: [Back to the Top](https://github.com/mikeroyal/Windows-11-Guide/blob/main/README.md#table-of-contents)


Question: How to: Node.js Learning Resources

Answer:
[Node.js](https://nodejs.org/) is a JavaScript runtime built on Chrome's V8 JavaScript engine that lets developers write command line tools and server-side scripts outside of a browser.

[Node.js Build Working Group](https://github.com/nodejs/build) maintains and controls infrastructure used for continuous integration (CI), releases, benchmarks, web hosting (of nodejs.org and other Node.js web properties) and more.

[The OpenJS Foundation](https://openjsf.org/) is made up of 32 open source JavaScript projects including Appium, Dojo, Electron, jQuery, Node.js, and webpack. The foundation's mission is to support the healthy growth of JavaScript and web technologies by providing a neutral organization to host and sustain projects, as well as collaboratively fund activities that benefit the ecosystem as a whole.

[Set up NodeJS on WSL 2](https://docs.microsoft.com/en-us/windows/nodejs/setup-on-wsl2)

[Getting started with Node.js in Google Cloud](https://cloud.google.com/nodejs/getting-started)

[Getting Started with Node.js in AWS](https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/getting-started-nodejs.html)

[Node.js App Hosting & Deployment in Microsoft Azure](https://azure.microsoft.com/en-us/develop/nodejs/)

[The Node.js best practices list ](https://github.com/goldbergyoni/nodebestpractices)

[Introduction to Node.js by W3Schools](https://www.w3schools.com/nodejs/nodejs_intro.asp)

[The Node.js Community Committee](https://github.com/nodejs/community-committee)

[Node.js Mentorship Program Initiative](https://github.com/nodejs/mentorship)

[Node.js tutorial in Visual Studio Code](https://code.visualstudio.com/docs/nodejs/nodejs-tutorial)

[Server-side Development with NodeJS, Express and MongoDB on Coursera](https://www.coursera.org/learn/server-side-nodejs)


Question: How to: Node.js Tools

Answer:
[NPM](https://www.npmjs.com/) is the company behind Node package manager, the npm Registry, and npm CLI.

[node-gyp](https://github.com/nodejs/node-gyp) is a cross-platform command-line tool written in Node.js for compiling native addon modules for Node.js. It contains a vendored copy of the gyp-next project that was previously used by the Chromium team, extended to support the development of Node.js native addons.

[nvm ](https://github.com/nvm-sh/nvm) is a version manager for node.js, designed to be installed per-user, and invoked per-shell. nvm works on any POSIX-compliant shell (sh, dash, ksh, zsh, bash), in particular on these platforms: unix, macOS, and windows WSL.

[node-docker](https://hub.docker.com/_/node/) is the official Node.js docker image, made with love by the node community.

[Mocha](https://github.com/mochajs/mocha) is a simple, flexible, fun JavaScript test framework for Node.js & The Browser.

[AVA](https://github.com/avajs/ava) is a test runner for Node.js with a concise API, detailed error output, embrace of new language features and process isolation that lets you develop with confidence.

[egg](https://eggjs.org/) is a born to build better enterprise frameworks and apps with Node.js & Koa.

[mysqljs](https://github.com/mysqljs/mysql) is a pure node.js JavaScript Client implementing the MySQL protocol.

[axios](https://github.com/axios/axios) is a promise based HTTP client for the browser and node.js.

[Fastify](https://www.fastify.io/) is a fast and low overhead web framework, for Node.js.

[Express](https://expressjs.com/) is a fast, unopinionated, minimalist web framework for node.

[Meteor](https://www.meteor.com/) is an ultra-simple environment for building modern web applications with JavavScript.

[NW.js](https://nwjs.io/) is an app runtime based on Chromium and node.js. You can write native apps in HTML and JavaScript with NW.js. It also lets you call Node.js modules directly from the DOM and enables a new way of writing native applications with all Web technologies.

[PM2](https://pm2.io/) is a production process manager for Node.js applications with a built-in load balancer. It allows you to keep applications alive forever, to reload them without downtime and to facilitate common system admin tasks.

[NestJS](https://nestjs.com/) is a framework for building efficient, scalable Node.js web applications. It uses modern JavaScript, is built with TypeScript and combines elements of OOP (Object Oriented Progamming), FP (Functional Programming), and FRP (Functional Reactive Programming).

[jenkins-nodejs](https://plugins.jenkins.io/nodejs/) is a Jenkins plugin for Node.js that provides the NodeJS auto-installer, allowing to create as many NodeJS installations "profiles" as you want.

[Strapi](https://strapi.io/) is an open source Node.js Headless CMS to easily build customisable APIs.

[Standard](https://standardjs.com/) is a JavaScript Style Guide, with linter & automatic code fixer.

[React Starter Kit](https://www.reactstarterkit.com/) is an isomorphic web app boilerplate for web development built on top of [Node.js](https://nodejs.org/), [Express](http://expressjs.com/), [GraphQL](http://graphql.org/) and [React](https://facebook.github.io/react/), containing modern web development tools such as [Webpack](https://webpack.github.io/), [Babel](https://babeljs.io/) and [Browsersync](https://www.browsersync.io/). Helping you to stay productive following the best practices.

[Hexo](https://hexo.io/) is a A fast, simple & powerful blog framework, powered by Node.js.


Question: How to: Networking

Answer: [Back to the Top](https://github.com/mikeroyal/Windows-11-Guide/blob/main/README.md#table-of-contents)


Question: How to: Networking Learning Resources

Answer:
[AWS Certified Security - Specialty Certification](https://aws.amazon.com/certification/certified-security-specialty/)

[Microsoft Certified: Azure Security Engineer Associate](https://docs.microsoft.com/en-us/learn/certifications/azure-security-engineer)

[Google Cloud Certified Professional Cloud Security Engineer](https://cloud.google.com/certification/cloud-security-engineer)

[Cisco Security Certifications](https://www.cisco.com/c/en/us/training-events/training-certifications/certifications/security.html)

[The Red Hat Certified Specialist in Security: Linux](https://www.redhat.com/en/services/training/ex415-red-hat-certified-specialist-security-linux-exam)

[Linux Professional Institute LPIC-3 Enterprise Security Certification](https://www.lpi.org/our-certifications/lpic-3-303-overview)

[Cybersecurity Training and Courses from IBM Skills](https://www.ibm.com/skills/topics/cybersecurity/)

[Cybersecurity Courses and Certifications by Offensive Security](https://www.offensive-security.com/courses-and-certifications/)

[Citrix Certified Associate – Networking(CCA-N)](http://training.citrix.com/cms/index.php/certification/networking/)

[Citrix Certified Professional – Virtualization(CCP-V)](https://www.globalknowledge.com/us-en/training/certification-prep/brands/citrix/section/virtualization/citrix-certified-professional-virtualization-ccp-v/)

[CCNP Routing and Switching](https://learningnetwork.cisco.com/s/ccnp-enterprise)

[Certified Information Security Manager(CISM)](https://www.isaca.org/credentialing/cism)

[Wireshark Certified Network Analyst (WCNA)](https://www.wiresharktraining.com/certification.html)

[Juniper Networks Certification Program Enterprise (JNCP)](https://www.juniper.net/us/en/training/certification/)

[Networking courses and specializations from Coursera](https://www.coursera.org/browse/information-technology/networking)

[Network & Security Courses from Udemy](https://www.udemy.com/courses/it-and-software/network-and-security/)

[Network & Security Courses from edX](https://www.edx.org/learn/cybersecurity)

  ## Tools & Networking Concepts

    • Connection: In networking, a connection refers to pieces of related information that are transferred through a network. This generally infers that a connection is built before the data transfer (by following the procedures laid out in a protocol) and then is deconstructed at the at the end of the data transfer.

    • Packet: A packet is, generally speaking, the most basic unit that is transferred over a network. When communicating over a network, packets are the envelopes that carry your data (in pieces) from one end point to the other.

Packets have a header portion that contains information about the packet including the source and destination, timestamps, network hops. The main portion of a packet contains the actual data being transferred. It is sometimes called the body or the payload.

    • Network Interface: A network interface can refer to any kind of software interface to networking hardware. For instance, if you have two network cards in your computer, you can control and configure each network interface associated with them individually.

A network interface may be associated with a physical device, or it may be a representation of a virtual interface. The "loop-back" device, which is a virtual interface to the local machine, is an example of this.

    • LAN: LAN stands for "local area network". It refers to a network or a portion of a network that is not publicly accessible to the greater internet. A home or office network is an example of a LAN.

    • WAN: WAN stands for "wide area network". It means a network that is much more extensive than a LAN. While WAN is the relevant term to use to describe large, dispersed networks in general, it is usually meant to mean the internet, as a whole.
If an interface is connected to the WAN, it is generally assumed that it is reachable through the internet.

    • Protocol: A protocol is a set of rules and standards that basically define a language that devices can use to communicate. There are a great number of protocols in use extensively in networking, and they are often implemented in different layers.

Some low level protocols are TCP, UDP, IP, and ICMP. Some familiar examples of application layer protocols, built on these lower protocols, are HTTP (for accessing web content), SSH, TLS/SSL, and FTP.

    • Port: A port is an address on a single machine that can be tied to a specific piece of software. It is not a physical interface or location, but it allows your server to be able to communicate using more than one application.

    • Firewall: A firewall is a program that decides whether traffic coming into a server or going out should be allowed. A firewall usually works by creating rules for which type of traffic is acceptable on which ports. Generally, firewalls block ports that are not used by a specific application on a server.

    • NAT: Network address translation is a


Question: How to: Interfaces

Answer: Interfaces are networking communication points for your computer. Each interface is associated with a physical or virtual networking device. Typically, your server will have one configurable network interface for each Ethernet or wireless internet card you have. In addition, it will define a virtual network interface called the "loopback" or localhost interface. This is used as an interface to connect applications and processes on a single computer to other applications and processes. You can see this referenced as the "lo" interface in many tools.


Question: How to: Virtualization

Answer:
[KVM (for Kernel-based Virtual Machine)](https://www.linux-kvm.org/page/Main_Page) is a full virtualization solution for Linux on x86 hardware containing virtualization extensions (Intel VT or AMD-V). It consists of a loadable kernel module, kvm.ko, that provides the core virtualization infrastructure and a processor specific module, kvm-intel.ko or kvm-amd.ko.

[QEMU](https://www.qemu.org) is a fast processor emulator using a portable dynamic translator. QEMU emulates a full system, including a processor and various peripherals. It can be used to launch a different Operating System without rebooting the PC or to debug system code.

[Hyper-V](https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/) enables running virtualized computer systems on top of a physical host. These virtualized systems can be used and managed just as if they were physical computer systems, however they exist in virtualized and isolated environment. Special software called a hypervisor manages access between the virtual systems and the physical hardware resources. Virtualization enables quick deployment of computer systems, a way to quickly restore systems to a previously known good state, and the ability to migrate systems between physical hosts.

[VirtManager](https://github.com/virt-manager/virt-manager) is a graphical tool for managing virtual machines via libvirt. Most usage is with QEMU/KVM virtual machines, but Xen and libvirt LXC containers are well supported. Common operations for any libvirt driver should work.

[oVirt](https://www.ovirt.org) is an open-source distributed virtualization solution, designed to manage your entire enterprise infrastructure. oVirt uses the trusted KVM hypervisor and is built upon several other community projects, including libvirt, Gluster, PatternFly, and Ansible.Founded by Red Hat as a community project on which Red Hat Enterprise Virtualization is based allowing for centralized management of virtual machines, compute, storage and networking resources, from an easy-to-use web-based front-end with platform independent access.

[Xen](https://github.com/xen-project/xen) is focused on advancing virtualization in a number of different commercial and open source applications, including server virtualization, Infrastructure as a Services (IaaS), desktop virtualization, security applications, embedded and hardware appliances, and automotive/aviation.

[Ganeti](https://github.com/ganeti/ganeti) is a virtual machine cluster management tool built on top of existing virtualization technologies such as Xen or KVM and other open source software. Once installed, the tool assumes management of the virtual instances (Xen DomU).

[Packer](https://www.packer.io/) is an open source tool for creating identical machine images for multiple platforms from a single source configuration. Packer is lightweight, runs on every major operating system, and is highly performant, creating machine images for multiple platforms in parallel. Packer does not replace configuration management like Chef or Puppet. In fact, when building images, Packer is able to use tools like Chef or Puppet to install software onto the image.

[Vagrant](https://www.vagrantup.com/) is a tool for building and managing virtual machine environments in a single workflow. With an easy-to-use workflow and focus on automation, Vagrant lowers development environment setup time, increases production parity, and makes the "works on my machine" excuse a relic of the past. It provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.

[VMware Workstation](https://www.vmware.com/products/workstation-pro.html) is a hosted hypervisor that runs on x64 versions of Windows and Linux operating systems; it enables users to set up virtual machines on a single physical machine, and use them simultaneously along with the actual machine.

[VirtualBox](https://www.virtualbox.org) is a powerful x86 and AMD64/Intel64 virtualization product for enterprise as well as home use. Not only is VirtualBox an extremely feature rich, high performance product for enterprise customers.


Question: How to: SQL/NoSQL Learning Resources

Answer:
[SQL](https://en.wikipedia.org/wiki/SQL) is a standard language for storing, manipulating and retrieving data in relational databases.

[NoSQL](https://www.ibm.com/cloud/blog/sql-vs-nosql) is a database that is interchangeably referred to as "nonrelational, or "non-SQL" to highlight that the database can handle huge volumes of rapidly changing, unstructured data in different ways than a relational (SQL-based) database with rows and tables.

[SQL Tutorial by W3Schools](https://www.w3schools.com/sql/)

[Learn SQL Skills Online from Coursera](https://www.coursera.org/courses?query=sql)

[SQL Courses Online from Udemy](https://www.udemy.com/topic/sql/)

[SQL Online Training Courses from LinkedIn Learning](https://www.linkedin.com/learning/topics/sql)

[Learn SQL For Free from Codecademy](https://www.codecademy.com/learn/learn-sql)

[GitLab's SQL Style Guide](https://about.gitlab.com/handbook/business-ops/data-team/platform/sql-style-guide/)

[OracleDB SQL Style Guide Basics](https://oracle.readthedocs.io/en/latest/sql/basics/style-guide.html)

[Tableau CRM: BI Software and Tools](https://www.salesforce.com/products/crm-analytics/overview/)

[Databases on AWS](https://aws.amazon.com/products/databases/)

[Best Practices and Recommendations for SQL Server Clustering in AWS EC2.](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/aws-sql-clustering.html)

[Connecting from Google Kubernetes Engine to a Cloud SQL instance.](https://cloud.google.com/sql/docs/mysql/connect-kubernetes-engine)

[Educational Microsoft Azure SQL resources](https://docs.microsoft.com/en-us/sql/sql-server/educational-sql-resources?view=sql-server-ver15)

[MySQL Certifications](https://www.mysql.com/certification/)

[SQL vs. NoSQL Databases: What's the Difference?](https://www.ibm.com/cloud/blog/sql-vs-nosql)

[What is NoSQL?](https://aws.amazon.com/nosql/)


Question: How to: SQL/NoSQL Tools and Databases

Answer:
[Azure Data Studio](https://github.com/Microsoft/azuredatastudio) is an open source data management tool that enables working with SQL Server, Azure SQL DB and SQL DW from Windows, macOS and Linux.

[Azure SQL Database](https://azure.microsoft.com/en-us/services/sql-database/)  is the intelligent, scalable, relational database service built for the cloud. It’s evergreen and always up to date, with AI-powered and automated features that optimize performance and durability for you. Serverless compute and Hyperscale storage options automatically scale resources on demand, so you can focus on building new applications without worrying about storage size or resource management.

[Azure SQL Managed Instance](https://azure.microsoft.com/en-us/services/azure-sql/sql-managed-instance/) is a fully managed SQL Server Database engine instance that's hosted in Azure and placed in your network. This deployment model makes it easy to lift and shift your on-premises applications to the cloud with very few application and database changes. Managed instance has split compute and storage components.

[Azure Synapse Analytics](https://azure.microsoft.com/en-us/services/synapse-analytics/) is a limitless analytics service that brings together enterprise data warehousing and Big Data analytics. It gives you the freedom to query data on your terms, using either serverless or provisioned resources at scale. It brings together the best of the SQL technologies used in enterprise data warehousing, Spark technologies used in big data analytics, and Pipelines for data integration and ETL/ELT.

[MSSQL for Visual Studio Code](https://marketplace.visualstudio.com/items?itemName=ms-mssql.mssql) is an extension for developing Microsoft SQL Server, Azure SQL Database and SQL Data Warehouse everywhere with a rich set of functionalities.

[SQL Server Data Tools (SSDT)](https://docs.microsoft.com/en-us/sql/ssdt/download-sql-server-data-tools-ssdt) is a development tool for building SQL Server relational databases, Azure SQL Databases, Analysis Services (AS) data models, Integration Services (IS) packages, and Reporting Services (RS) reports. With SSDT, a developer can design and deploy any SQL Server content type with the same ease as they would develop an application in Visual Studio or Visual Studio Code.

[Bulk Copy Program](https://docs.microsoft.com/en-us/sql/tools/bcp-utility) is a command-line tool that comes with Microsoft SQL Server. BCP, allows you to import and export large amounts of data in and out of SQL Server databases quickly snd efficeiently.

[SQL Server Migration Assistant](https://www.microsoft.com/en-us/download/details.aspx?id=54258) is a tool from Microsoft that simplifies database migration process from Oracle to SQL Server, Azure SQL Database, Azure SQL Database Managed Instance and Azure SQL Data Warehouse.

[SQL Server Integration Services](https://docs.microsoft.com/en-us/sql/integration-services/sql-server-integration-services?view=sql-server-ver15) is a development platform for building enterprise-level data integration and data transformations solutions. Use Integration Services to solve complex business problems by copying or downloading files, loading data warehouses, cleansing and mining data, and managing SQL Server objects and data.

[SQL Server Business Intelligence(BI)](https://www.microsoft.com/en-us/sql-server/sql-business-intelligence) is a collection of tools in Microsoft's SQL Server for transforming raw data into information businesses can use to make decisions.

[Tableau](https://www.tableau.com/) is a Data Visualization software used in relational databases, cloud databases, and spreadsheets. Tableau was acquired by [Salesforce in August 2019](https://investor.salesforce.com/press-releases/press-release-details/2019/Salesforce-Completes-Acquisition-of-Tableau/default.aspx).

[DataGrip](https://www.jetbrains.com/datagrip/) is a professional DataBase IDE developed by Jet Brains that provides context-sensitive code completion, helping you to write SQL code faster. Completion is aware of the tables structure, foreign keys, and even database objects created in code you're editing.

[RStudio](https://rstudio.com/) is an integrated development environment for R and Python, with a console, syntax-highlighting editor that supports direct code execution, and tools for plotting, history, debugging and workspace management.

[MySQL](https://www.mysql.com/) is a fully managed database service to deploy cloud-native applications using the world's most popular open source database.

[PostgreSQL](https://www.postgresql.org/) is a powerful, open source object-relational database system with over 30 years of active development that has earned it a strong reputation for reliability, feature robustness, and performance.

[Amazon DynamoDB](https://aws.amazon.com/dynamodb/) is a key-value and document database that delivers single-digit millisecond performance at any scale. It is a fully managed, multiregion, multimaste


Question: How to: Contribute

Answer: - [x] If would you like to contribute to this guide simply make a [Pull Request](https://github.com/mikeroyal/Windows-11-Guide/pulls).


Question: How to: Pre-requisites

Answer: Make sure Docker and [Docker Compose](https://docs.docker.com/compose/install/) are installed on your Docker host machine.


Question: How to: Quick Start

Answer:
```
git clone https://github.com/geerlingguy/internet-monitoring
cd internet-monitoring
docker-compose up -d
```

Go to [http://localhost:3030/d/o9mIe_Aik/internet-connection](http://localhost:3030/d/o9mIe_Aik/internet-connection) (change `localhost` to your docker host ip/name).


Question: How to: Configuration

Answer:
To change what hosts you ping you change the `targets` section in [/prometheus/pinghosts.yaml](./prometheus/pinghosts.yaml) file.

For speedtest the only relevant configuration is how often you want the check to happen. It is at 30 minutes by default which might be too much if you have limit on downloads. This is changed by editing `scrape_interval` under `speedtest` in [/prometheus/prometheus.yml](./prometheus/prometheus.yml).

Once configurations are done, run the following command:

    $ docker-compose up -d

That's it. docker-compose builds the entire Grafana and Prometheus stack automagically.

The Grafana Dashboard is now accessible via: `http://
:3030` for example http://localhost:3030

username - admin
password - wonka (Password is stored in the `config.monitoring` env file)

The DataSource and Dashboard for Grafana are automatically provisioned.

If all works it should be available at http://localhost:3030/d/o9mIe_Aik/internet-connection - if no data shows up try change the timeduration to something smaller.


Question: How to: Interesting urls

Answer:
http://localhost:9090/targets shows status of monitored targets as seen from prometheus - in this case which hosts being pinged and speedtest. note: speedtest will take a while before it shows as UP as it takes about 30s to respond.

http://localhost:9090/graph?g0.expr=probe_http_status_code&g0.tab=1 shows prometheus value for `probe_http_status_code` for each host. You can edit/play with additional values. Useful to check everything is okey in prometheus (in case Grafana is not showing the data you expect).

http://localhost:9115 blackbox exporter endpoint. Lets you see what have failed/succeded.

http://localhost:9798/metrics speedtest exporter endpoint. Does take about 30 seconds to show its result as it runs an actual speedtest when requested.


Question: How to: Thanks and a disclaimer

Answer:
Thanks to @maxandersen for making the original project this fork is based on.

Thanks to @vegasbrianc work on making a [super easy docker](https://github.com/vegasbrianc/github-monitoring) stack for running prometheus and grafana.

This setup is not secured in any way, so please only use on non-public networks, or find a way to secure it on your own.


Question: How to: Table of contents

Answer:
- [FastAPI with Observability](#fastapi-with-observability)
  - [Table of contents](#table-of-contents)
  - [Quick Start](#quick-start)
  - [Explore with Grafana](#explore-with-grafana)
    - [Metrics to Traces](#metrics-to-traces)
    - [Traces to Logs](#traces-to-logs)
    - [Logs to Traces](#logs-to-traces)
  - [Detail](#detail)
    - [FastAPI Application](#fastapi-application)
      - [Traces and Logs](#traces-and-logs)
      - [Span Inject](#span-inject)
      - [Metrics](#metrics)
      - [OpenTelemetry Instrumentation](#opentelemetry-instrumentation)
    - [Prometheus - Metrics](#prometheus---metrics)
      - [Prometheus Config](#prometheus-config)
      - [Grafana Data Source](#grafana-data-source)
    - [Tempo - Traces](#tempo---traces)
      - [Grafana Data Source](#grafana-data-source-1)
    - [Loki - Logs](#loki---logs)
      - [Loki Docker Driver](#loki-docker-driver)
      - [Grafana Data Source](#grafana-data-source-2)
    - [Grafana](#grafana)
  - [Reference](#reference)


Question: How to: Explore with Grafana

Answer:
Grafana provides a great solution, which could observe specific actions in service between traces, metrics, and logs through trace ID and exemplar.

![Observability Correlations](./images/observability-correlations.jpeg)

Image Source: [Grafana](https://grafana.com/blog/2021/03/31/intro-to-exemplars-which-enable-grafana-tempos-distributed-tracing-at-massive-scale/)


Question: How to: Metrics to Traces

Answer:
Get Trace ID from an exemplar in metrics, then query in Tempo.

Query: `histogram_quantile(.99,sum(rate(fastapi_requests_duration_seconds_bucket{app_name="app-a", path!="/metrics"}[1m])) by(path, le))`

![Metrics to Traces](./images/metrics-to-traces.png)


Question: How to: Traces to Logs

Answer:
Get Trace ID and tags (here is `service.name`) defined in Tempo data source from span, then query with Loki.

![Traces to Logs](./images/traces-to-logs.png)


Question: How to: FastAPI Application

Answer:
For a more complex scenario, we use three FastAPI applications with the same code in this demo. There is a cross-service action in `/chain` endpoint, which provides a good example of how to use OpenTelemetry SDK and how Grafana presents trace information.

#### Traces and Logs

We use [OpenTelemetry Python SDK](https://github.com/open-telemetry/opentelemetry-python) to send trace info with gRCP to Tempo. Each request span contains other child spans when using OpenTelemetry instrumentation. The reason is that instrumentation will catch each internal asgi interaction ([opentelemetry-python-contrib issue #831](https://github.com/open-telemetry/opentelemetry-python-contrib/issues/831#issuecomment-1005163018)). If you want to get rid of the internal spans, there is a [workaround](https://github.com/open-telemetry/opentelemetry-python-contrib/issues/831#issuecomment-1116225314) in the same issue #831 by using a new OpenTelemetry middleware with two overridden methods for span processing.

We use [OpenTelemetry Logging Instrumentation](https://opentelemetry-python-contrib.readthedocs.io/en/latest/instrumentation/logging/logging.html) to override the logger format with another format with trace id and span id.

```py


Question: How to: fastapi_app/utils.py

Answer:
def setting_otlp(app: ASGIApp, app_name: str, endpoint: str, log_correlation: bool = True) -> None:
    # Setting OpenTelemetry
    # set the service name to show in traces
    resource = Resource.create(attributes={
        "service.name": app_name # for Tempo to distinguish source
    })

    # set the tracer provider
    tracer = TracerProvider(resource=resource)
    trace.set_tracer_provider(tracer)

    tracer.add_span_processor(BatchSpanProcessor(
        OTLPSpanExporter(endpoint=endpoint)))

    if log_correlation:
        LoggingInstrumentor().instrument(set_logging_format=True)

    FastAPIInstrumentor.instrument_app(app, tracer_provider=tracer)
```

The following image shows the span info sent to Tempo and queried on Grafana. Trace span info provided by `FastAPIInstrumentor` with trace ID (ef106bd57e5d8223a23ee9e3c93be73b), span id(b4fff543781f9beb), service name(app-a), custom attributes(service.name=app-a) and so on.

![Span Information](./images/span-info.png)

Log format with trace id and span id, which is overridden by `LoggingInstrumentor``

```txt
%(asctime)s %(levelname)s [%(name)s] [%(filename)s:%(lineno)d] [trace_id=%(otelTraceID)s span_id=%(otelSpanID)s resource.service.name=%(otelServiceName)s] - %(message)s
```

The following image is what the logs look like.

![Log With Trace ID And Span ID](./images/log-format.png)

#### Span Inject

If you want other services to use the same Trace ID, you have to use `inject` function to add current span information to the header. Because OpenTelemetry FastAPI instrumentation only takes care of the asgi app's request and response, it does not affect any other modules or actions like sending HTTP requests to other servers or function calls.

```py


Question: How to: fastapi_app/main.py

Answer:
from opentelemetry.propagate import inject

@app.get("/chain")
async def chain(response: Response):

    headers = {}
    inject(headers)  # inject trace info to header

    async with httpx.AsyncClient() as client:
        await client.get(f"http://localhost:8000/", headers=headers,)
    async with httpx.AsyncClient() as client:
        await client.get(f"http://{TARGET_ONE_HOST}:8000/io_task", headers=headers,)
    async with httpx.AsyncClient() as client:
        await client.get(f"http://{TARGET_TWO_HOST}:8000/cpu_task", headers=headers,)

    return {"path": "/chain"}
```

Alternatively, we can use the [instrumentation library for HTTPX](https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation/opentelemetry-instrumentation-httpx) to instrument HTTPX. Following is the example of using OpenTelemetry HTTPX Instrumentation which will automatically inject trace info to the header.

```py
import httpx
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor

HTTPXClientInstrumentor().instrument()

@app.get("/chain")
async def chain(response: Response):
    async with httpx.AsyncClient() as client:
        await client.get(f"http://localhost:8000/")
    async with httpx.AsyncClient() as client:
        await client.get(f"http://{TARGET_ONE_HOST}:8000/io_task")
    async with httpx.AsyncClient() as client:
        await client.get(f"http://{TARGET_TWO_HOST}:8000/cpu_task")

    return {"path": "/chain"}
```

#### Metrics

Use [Prometheus Python Client](https://github.com/prometheus/client_python) to generate OpenTelemetry format metric with [exemplars](https://github.com/prometheus/client_python#exemplars) and expose on `/metrics` for Prometheus.

In order to add an exemplar to metrics, we retrieve the trace id from the current span for the exemplar and add the trace id dict to the Histogram or Counter metrics.

```py


Question: How to: retrieve trace id for exemplar

Answer:
span = trace.get_current_span()
trace_id = trace.format_trace_id(
      span.get_span_context().trace_id)

REQUESTS_PROCESSING_TIME.labels(method=method, path=path, app_name=self.app_name).observe(
      after_time - before_time, exemplar={'TraceID': trace_id}
)
```

Because exemplars is a new datatype proposed in [OpenMetrics](https://github.com/prometheus/OpenMetrics/blob/main/specification/OpenMetrics.md#exemplars), `/metrics` have to use `CONTENT_TYPE_LATEST` and `generate_latest` from `prometheus_client.openmetrics.exposition` module instead of `prometheus_client` module ([Prometheus Client Document](https://prometheus.github.io/client_python/instrumenting/exemplars/)). Otherwise using the wrong generate_latest the exemplars dict behind Counter and Histogram will never show up, and using the wrong CONTENT_TYPE_LATEST will cause Prometheus scraping to fail.

```py


Question: How to: Prometheus - Metrics

Answer:
Collects metrics from applications.

#### Prometheus Config

Define all FastAPI applications metrics scrape jobs in `etc/prometheus/prometheus.yml`.

```yaml
...
scrape_configs:
  - job_name: 'app-a'
    scrape_interval: 5s
    static_configs:
      - targets: ['app-a:8000']
  - job_name: 'app-b'
    scrape_interval: 5s
    static_configs:
      - targets: ['app-b:8000']
  - job_name: 'app-c'
    scrape_interval: 5s
    static_configs:
      - targets: ['app-c:8000']
```

#### Grafana Data Source

Add an Exemplars which uses the value of `TraceID` label to create a Tempo link.

Grafana data source setting example:

![Data Source of Prometheus: Exemplars](./images/prometheus-exemplars.png)

Grafana data sources config example:

```yaml
name: Prometheus
type: prometheus
typeName: Prometheus
access: proxy
url: http://prometheus:9090
password: ''
user: ''
database: ''
basicAuth: false
isDefault: true
jsonData:
exemplarTraceIdDestinations:
   - datasourceUid: tempo
      name: TraceID
httpMethod: POST
readOnly: false
editable: true
```


Question: How to: Tempo - Traces

Answer:
Receives spans from applications.

#### Grafana Data Source

[Trace to logs](https://grafana.com/docs/grafana/latest/datasources/tempo/#trace-to-logs) setting:

1. Data source: target log source
2. Tags: key of tags from the trace, which will be log query criteria if the key exists in the trace
   1. Without as value: use the key as query label
   2. With as value: convert the tag key to another key as query label

Grafana data source setting example:

![Data Source of Tempo: Trace to logs](./images/tempo-trace-to-logs.png)

Grafana data sources config example:

```yaml
uid: tempo
orgId: 1
name: Tempo
type: tempo
typeName: Tempo
access: proxy
url: http://tempo
password: ''
user: ''
database: ''
basicAuth: false
isDefault: false
jsonData:
  nodeGraph:
    enabled: true
  tracesToLogsV2:
    customQuery: false
    datasourceUid: loki
    filterBySpanID: false
    filterByTraceID: true
    tags:
      - key: service.name
        value: compose_service
readOnly: false
editable: true
```


Question: How to: Loki - Logs

Answer:
Collect logs with Loki Docker Driver from all services.

#### Loki Docker Driver

1. Use [YAML anchor and alias](https://support.atlassian.com/bitbucket-cloud/docs/yaml-anchors/) feature to set logging options for each service.
2. Set [Loki Docker Driver options](https://grafana.com/docs/loki/latest/clients/docker-driver/configuration/)
   1. loki-url: loki service endpoint
   2. loki-pipeline-stages: processes multiline log from FastAPI application with multiline and regex stages ([reference](https://grafana.com/docs/loki/latest/clients/promtail/stages/multiline/))

```yaml
x-logging: &default-logging # anchor(&): 'default-logging' for defines a chunk of configuration
  driver: loki
  options:
    loki-url: 'http://localhost:3100/api/prom/push'
    loki-pipeline-stages: |
      - multiline:
          firstline: '^\d{4}-\d{2}-\d{2} \d{1,2}:\d{2}:\d{2}'
          max_wait_time: 3s
      - regex:
          expression: '^(?P
\d{4}-\d{2}-\d{2} \d{1,2}:\d{2}:\d{2},\d{3}) (?P
(?s:.*))$$'


Question: How to: Use $$ (double-dollar sign) when your configuration needs a literal dollar sign.

Answer:
services:
   foo:
      image: foo
      logging: *default-logging # alias(*): refer to 'default-logging' chunk
```

#### Grafana Data Source

Add a TraceID derived field to extract the trace id and create a Tempo link from the trace id.

Grafana data source setting example:

![Data Source of Loki: Derived fields](./images/loki-derive-fields.png)

Grafana data source config example:

```yaml
name: Loki
type: loki
typeName: Loki
access: proxy
url: http://loki:3100
password: ''
user: ''
database: ''
basicAuth: false
isDefault: false
jsonData:
  derivedFields:
    - datasourceUid: tempo
      matcherRegex: (?:trace_id)=(\w+)
      matcherType: regex
      name: TraceID
      url: $${__value.raw}
      # Use $$ (double-dollar sign) when your configuration needs a literal dollar sign.
readOnly: false
editable: true
```


Question: How to: grafana in docker-compose.yaml

Answer:
grafana:
   image: grafana/grafana:12.0.0
   volumes:
      - ./etc/grafana/:/etc/grafana/provisioning/datasources # data sources
      - ./etc/dashboards.yaml:/etc/grafana/provisioning/dashboards/dashboards.yaml # dashboard setting
      - ./etc/dashboards:/etc/grafana/dashboards # dashboard json files directory
```


Question: How to: Altinity Grafana datasource plugin for ClickHouse® (grafana Grafana 4.6+ supported)

Answer:
Altinity ClickHouse datasource plugin provides a support for [ClickHouse](https://clickhouse.tech) as a backend database.

Initially plugin developed by Vertamedia, maintaned by Altinity since 2020.


Question: How to: setup notes for plugin version 3.4.9+

Answer:
Because we added `output_format_json_quote_64bit_integers=1` to any browser request from plugin,
if your user which you use for connection have readonly=1, change it to readonly=2, for example
`/etc/clickhouse-server/users.d/readonly.xml`
```xml
2
```


Question: How to: ClickHouse 25.8+ setup notes for plugin version before 3.4.9

Answer:
New ClickHouse version change default behavior for output_format_json_quote_64bit_integers=0,
which not allows to read generated JSON in JavaScript, look details in https://github.com/ClickHouse/ClickHouse/issues/86553
To properly works old version Altinity clickhouse datasource plugin for Grafana use following config
`/etc/clickhouse-server/users.d/output_format_json_quote_64bit_integers.xml`
```xml
1
```


Question: How to: Grafana 10+ setup notes for plugin version before 3.0.0

Answer:
Old versions of Altinity ClickHouse datasource plugin for Grafana written in Angular. So you can watch warning like
```
Angular plugin
This data source plugin uses a deprecated, legacy platform based on AngularJS and will stop working in future releases of Grafana.
```

Don't worry about warning message, plugin will still working until Grafana 11 will release, after it upgrade to Altinity ClickHouse datasource plugin for Grafana to 3.x version is required.


Question: How to: Grafana 7+ setup notes for plugin version before 2.2.0

Answer:
When 2.0.x and 2.1.x vertamedia-clickhouse-grafana plugin versions released Grafana team didn't provide worked signing method for community plugins.
Current sign process describe on [grafana.com](https://grafana.com/docs/grafana/latest/developers/plugins/sign-a-plugin/)

so, for properly setup 2.0.x and 2.1.x plugins you need change configuration option

```ini
[plugins]
allow_loading_unsigned_plugins=vertamedia-clickhouse-datasource
```

or setup environment variable

```bash
GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=vertamedia-clickhouse-datasource
```

You can install plugin from [grafana.com](https://grafana.com/plugins/vertamedia-clickhouse-datasource)

OR

Copy files to your [Grafana plugin directory](https://grafana.com/docs/grafana/latest/plugins/installation/#install-plugin-on-local-grafana).
Restart Grafana, check data sources list at Configuration -> Datasources -> New, choose ClickHouse option.

![Datasources](https://github.com/Altinity/clickhouse-grafana/raw/master/.github/images/01_data_sources.png)
![Add new datasource](https://github.com/Altinity/clickhouse-grafana/raw/master/.github/images/02_add_data_source.png)
![Datasource types](https://github.com/Altinity/clickhouse-grafana/raw/master/.github/images/03_filter_click_to_plugin.png)


Question: How to: Access to ClickHouse via HTTP / HTTPS

Answer:
Page configuration is standard

![settings](https://github.com/Altinity/clickhouse-grafana/raw/master/.github/images/04_datasource_settings.png)

There is a small feature - ClickHouse treats HTTP Basic Authentication credentials as a database user and will try to run queries using its name.


Question: How to: [CHProxy](https://github.com/ContentSquare/chproxy) (optional)

Answer:
Using of [CHProxy](https://github.com/ContentSquare/chproxy) will bring additional features:

* Easily setup `HTTPS` access to ClickHouse as shown [here](https://github.com/ContentSquare/chproxy#authorize-users-by-passwords-via-https)
to provide secure access.
* Limit concurrency and execution time for requests from `Grafana` as shown [here](https://github.com/ContentSquare/chproxy#spread-selects-from-reporting-apps-among-cluster-nodes)
to prevent `ClickHouse` overloading from `Grafana`.
* Protection against request bursts for dashboards with numerous graphs. `CHProxy` allows queueing requests and execute them sequentially.
To learn more - read about params `max_queue_size` and `max_queue_time` at [CHProxy](https://github.com/ContentSquare/chproxy) page.
* Response caching for the most frequent queries as shown [here](https://github.com/ContentSquare/chproxy#caching).

`Caching` will protect `ClickHouse` from excessive refreshes and will be optimal option for popular dashboards.
> Hint - if you need to cache requests like `last 24h` where timestamp changes constantly then try to use `Round` option at `Raw Editor`


Question: How to: Query setup

Answer:
Query setup interface:

![query editor image](https://github.com/Altinity/clickhouse-grafana/raw/master/.github/images/05_query_settings.png)

First row `FROM` contains two options: database and table. Table values depends on a selected database.
Next rows contains selectors for time filtering:

Column timestamp time
* DateTime ([DateTime](https://clickhouse.com/docs/en/sql-reference/data-types/datetime/))
* DateTime64 ([DateTime64](https://clickhouse.com/docs/en/sql-reference/data-types/datetime64/))
* TimeStamp ([UInt32](https://clickhouse.com/docs/en/sql-reference/data-types/int-uint/)).

Timestamp column
Date column

> `Timestmap column` are required for time-based macros and functions because all analytics based on these values.
> Plugin will try to detect Date, Date32 column automatically

Button `Go to Query` is just a toggler to Raw SQL Editor


Question: How to: Raw SQL Editor

Answer:
Raw Editor allows custom SQL queries to be written:

![raw editor image](https://github.com/Altinity/clickhouse-grafana/raw/master/.github/images/06_raw_sql_editor.png)

Raw Editor allows typing queries, get info about functions and macros, format queries as Clickhouse do.
To Execute query on server press "Run Query" or just leave focus from SQL editor textarea.

Under the Editor you can find options which allows setup rounding, time column step
and `Add metadata` to SQL query which allows know which dashboard and user produce workload to your ClickHouse server.

Press `Show Generated SQL` for see a raw query (all macros and functions have already been replaced) which will be sent directly to ClickHouse.
![generated sql](https://github.com/Altinity/clickhouse-grafana/raw/master/.github/images/07_generated_sql.png)


Question: How to: Macros support

Answer:
Plugin supports the following marcos:

* $table - replaced with selected table name from Query Builder
* $dateCol - replaced with `Column:Date` value from Query Builder
* $dateTimeCol - replaced with `Column:DateTime` or `Column:TimeStamp` value from Query Builder
* $from - replaced with (timestamp with ms)/1000 value of UI selected "Time Range:From"
* $to - replaced with (timestamp with ms)/1000 value of UI selected "Time Range:To"
* $interval - replaced with selected "Group by a time interval" value (as a number of seconds)
* $timeFilter - replaced with currently selected "Time Range".
  Requires Column:Date and Column:DateTime or Column:TimeStamp to be selected.
* $timeFilterByColumn($column) - replaced with currently selected "Time Range" for a column passed as `$column` argument. Use it in queries or query variables as `...WHERE $timeFilterColumn($column)...` or `...WHERE $timeFilterColumn(created_at)...`.
* $timeSeries - replaced with special ClickHouse construction to convert results as time-series data. Use it as "SELECT $timeSeries...".
* $naturalTimeSeries - replaced with special ClickHouse construction to convert results as time-series with in a logical/natural breakdown. Use it as "SELECT $naturalTimeSeries...".
* $unescape - unescapes variable value by removing single quotes. Used for multiple-value string variables: "SELECT $unescape($column) FROM requests WHERE $unescape($column) = 5"
* $adhoc - replaced with a rendered ad-hoc filter expression, or "1" if no ad-hoc filters exist. Since ad-hoc applies automatically only to outer queries the macros can be used for filtering in inner queries.

A description of macros is available by typing their names in Raw Editor


Question: How to: $rate(cols...) - converts query results as "change rate per interval"

Answer:
Example usage:

```sql
$rate(countIf(Type = 200) AS good, countIf(Type != 200) AS bad) FROM requests
```

Query will be transformed into:

```sql
SELECT
    t,
    good / runningDifference(t / 1000) AS goodRate,
    bad / runningDifference(t / 1000) AS badRate
FROM
(
    SELECT
        (intDiv(toUInt32(EventTime), 60)) * 1000 AS t,
        countIf(Type = 200) AS good,
        countIf(Type != 200) AS bad
    FROM requests
    WHERE ((EventDate >= toDate(1482796747)) AND (EventDate <= toDate(1482853383))) AND ((EventTime >= toDateTime(1482796747)) AND (EventTime <= toDateTime(1482853383)))
    GROUP BY t
    ORDER BY t
)
```

---


Question: How to: $columns(key, value) - query values as array of [key, value], where key will be used as label

Answer:
Example usage:

```sql
$columns(OSName, count(*) c)
FROM requests
INNER JOIN oses USING (OS)
```

Query will be transformed into:

```sql
SELECT
    t,
    groupArray((OSName, c)) AS groupArr
FROM
(
    SELECT
        (intDiv(toUInt32(EventTime), 60) * 60) * 1000 AS t,
        OSName,
        count(*) AS c
    FROM requests
    INNER JOIN oses USING (OS)
    WHERE ((EventDate >= toDate(1482796627)) AND (EventDate <= toDate(1482853383))) AND ((EventTime >= toDateTime(1482796627)) AND (EventTime <= toDateTime(1482853383)))
    GROUP BY
        t,
        OSName
    ORDER BY
        t,
        OSName
)
GROUP BY t
ORDER BY t
```

This will help to build the next graph:

![req_by_os image](https://github.com/Altinity/clickhouse-grafana/raw/master/.github/images/08_requests_by_os.png)

---


Question: How to: $columnsMs(key, value) - same as $columns but for time series with ms

Answer:
Example usage:

```sql
$columnsMs(OSName, count(*) c)
FROM requests
INNER JOIN oses USING (OS)
```

Query will be transformed into:

```sql
SELECT
    t,
    groupArray((OSName, c)) AS groupArr
FROM
(
    SELECT
        $timeSeriesMs AS t,
        OSName,
        count(*) AS c
    FROM requests
    INNER JOIN oses USING (OS)
    WHERE ((EventDate >= toDate(1482796627)) AND (EventDate <= toDate(1482853383))) AND ((EventTime >= toDateTime64(1482796627,3)) AND (EventTime <= toDateTime64(1482853383,3)))
    GROUP BY
        t,
        OSName
    ORDER BY
        t,
        OSName
)
GROUP BY t
ORDER BY t
```

---


Question: How to: $lttb(buckets_number, [field1, ... fieldN], x_field, y_field) - allow show down-sampled time series which will contains more outliers than avg or other kind of aggregation

Answer:
If bucket_number is `auto`, then it will calculated as `toUInt64( ($to-$from) / $interval )`
Example usage:

```sql
$lttb(auto, category, event_time, count(*) c)
FROM requests GROUP BY category
```

Query will be transformed into:

```sql
SELECT category, lttb_result.1 AS event_time, lttb_result.2 AS c FROM (
    SELECT category, untuple(arrayJoin(lttb(toUInt64( ($to - $from) / $interval ))(event_time, cont(*) AS c))) AS lttb_result
    FROM requests WHERE $timeFilter GROUP BY category
) ORDER BY event_time
```

---

---


Question: How to: $rateColumns(key, value) - is a combination of $columns and $rate

Answer:
Example usage:

```sql
$rateColumns(OS, count(*) c) FROM requests
```

Query will be transformed into:

```sql
SELECT
    t,
    arrayMap(lambda(tuple(a), (a.1, a.2 / runningDifference(t / 1000))), groupArr)
FROM
(
    SELECT
        t,
        groupArray((OS, c)) AS groupArr
    FROM
    (
        SELECT
            (intDiv(toUInt32(EventTime), 60) * 60) * 1000 AS t,
            OS,
            count(*) AS c
        FROM requests
        WHERE ((EventDate >= toDate(1482796867)) AND (EventDate <= toDate(1482853383))) AND ((EventTime >= toDateTime(1482796867)) AND (EventTime <= toDateTime(1482853383)))
        GROUP BY
            t,
            OS
        ORDER BY
            t,
            OS
    )
    GROUP BY t
    ORDER BY t
)

```

---


Question: How to: $rateColumnsAggregated(key, subkey, aggFunction1, value1, ... aggFunctionN, valueN) - if you need calculate `rate` for higher cardinality dimension and then aggregate by lower cardinality dimension

Answer:
Example usage:

```sql
$rateColumnsAggregated(datacenter, concat(datacenter,interface) AS dc_interface, sum, tx_bytes * 1014 AS tx_kbytes, sum, max(rx_bytes) AS rx_bytes) FROM traffic
```

Query will be transformed into:

```sql
SELECT
    t,
    datacenter,
    sum(tx_kbytesRate) AS tx_bytesRateAgg,
    sum(rx_bytesRate) AS rx_bytesRateAgg
FROM
(
    SELECT
        t,
        datacenter,
        dc_interface,
        tx_kbytes / runningDifference(t / 1000) AS tx_kbytesRate,
        rx_bytes / runningDifference(t / 1000) AS rx_bytesRate
    FROM
    (
        SELECT
            (intDiv(toUInt32(event_time), 60) * 60) * 1000 AS t,
            datacenter,
            concat(datacenter,interface) AS dc_interface,
            max(tx_bytes * 1024) AS tx_kbytes,
            max(rx_bytes) AS rx_bytes
        FROM traffic
        WHERE ((event_date >= toDate(1482796867)) AND (event_date <= toDate(1482853383)))
          AND ((event_time >= toDateTime(1482796867)) AND (event_time <= toDateTime(1482853383)))
        GROUP BY
            t,
            datacenter,
            dc_interface
        ORDER BY
            t,
            datacenter,
            dc_interface
    )
)
GROUP BY
  t,
  datacenter
ORDER BY
  datacenter,
  t
```

look [issue 386](https://github.com/Altinity/clickhouse-grafana/issues/386) for reasons for implementation

---


Question: How to: $perSecond(cols...) - converts query results as "change rate per interval" for Counter-like(growing only) metrics

Answer:
Example usage:

```sql
$perSecond(Requests) FROM requests
```

Query will be transformed into:

```sql
SELECT
    t,
    if(runningDifference(max_0) < 0, nan, runningDifference(max_0) / runningDifference(t / 1000)) AS max_0_PerSecond
FROM
(
    SELECT
        (intDiv(toUInt32(EventTime), 60) * 60) * 1000 AS t,
        max(Requests) AS max_0
    FROM requests
    WHERE ((EventDate >= toDate(1535711819)) AND (EventDate <= toDate(1535714715)))
    AND ((EventTime >= toDateTime(1535711819)) AND (EventTime <= toDateTime(1535714715)))
    GROUP BY t
    ORDER BY t
)
```

// see [issue 78](https://github.com/Altinity/clickhouse-grafana/issues/78) for the background

---


Question: How to: $perSecondColumns(key, value) - is a combination of $columns and $perSecond for Counter-like metrics

Answer:
Example usage:

```sql
$perSecondColumns(Protocol, Requests) FROM requests WHERE Protocol in ('udp','tcp')
```

Query will be transformed into:

```sql
SELECT
    t,
    groupArray((perSecondColumns, max_0_PerSecond)) AS groupArr
FROM
(
    SELECT
        t,
        Protocol,
        if(runningDifference(max_0) < 0 OR neighbor(perSecondColumns,-1,perSecondColumns) != perSecondColumns, nan, runningDifference(max_0) / runningDifference(t / 1000)) AS max_0_PerSecond
    FROM
    (
        SELECT
            (intDiv(toUInt32(EventTime), 60) * 60) * 1000 AS t,
            Protocol AS perSecondColumns,
            max(Requests) AS max_0
        FROM requests
        WHERE ((EventDate >= toDate(1535711819)) AND (EventDate <= toDate(1535714715)))
        AND ((EventTime >= toDateTime(1535711819)) AND (EventTime <= toDateTime(1535714715)))
        AND (Protocol IN ('udp', 'tcp'))
        GROUP BY
            t,
            Protocol
        ORDER BY
            t,
            Protocol
    )
)
GROUP BY t
ORDER BY t
```

// see [issue 80](https://github.com/Altinity/clickhouse-grafana/issues/80) for the background

---


Question: How to: $delta(cols...) - converts query results as "delta value inside interval" for Counter-like(growing only) metrics, will negative if counter reset

Answer:
Example usage:

```sql
$delta(Requests) FROM requests
```

Query will be transformed into:

```sql
SELECT
    t,
    runningDifference(max_0) AS max_0_Delta
FROM
(
    SELECT
        (intDiv(toUInt32(EventTime), 60) * 60) * 1000 AS t,
        max(Requests) AS max_0
    FROM requests
    WHERE ((EventDate >= toDate(1535711819)) AND (EventDate <= toDate(1535714715)))
    AND ((EventTime >= toDateTime(1535711819)) AND (EventTime <= toDateTime(1535714715)))
    GROUP BY t
    ORDER BY t
)
```

// see [issue 455](https://github.com/Altinity/clickhouse-grafana/issues/455) for the background

---


Question: How to: $deltaColumns(key, value) - is a combination of $columns and $delta for Counter-like metrics

Answer:
Example usage:

```sql
$deltaColumns(Protocol, Requests) FROM requests WHERE Protocol in ('udp','tcp')
```

Query will be transformed into:

```sql
SELECT
    t,
    groupArray((deltaColumns, max_0_Delta)) AS groupArr
FROM
(
    SELECT
        t,
        deltaColumns,
        if (neighbor(deltaColumns,-1,deltaColumns) != deltaColumns, 0, runningDifference(max_0)) AS max_0_Delta
    FROM
    (
        SELECT
            (intDiv(toUInt32(EventTime), 60) * 60) * 1000 AS t,
            Protocol AS deltaColumns,
            max(Requests) AS max_0
        FROM requests
        WHERE ((EventDate >= toDate(1535711819)) AND (EventDate <= toDate(1535714715)))
        AND ((EventTime >= toDateTime(1535711819)) AND (EventTime <= toDateTime(1535714715)))
        AND (Protocol IN ('udp', 'tcp'))
        GROUP BY
            t,
            Protocol
        ORDER BY
            t,
            Protocol
    )
)
GROUP BY t
ORDER BY t
```

// see [issue 455](https://github.com/Altinity/clickhouse-grafana/issues/455) for the background

---


Question: How to: $increase(cols...) - converts query results as "non-negative delta value inside interval" for Counter-like(growing only) metrics, will zero if counter reset and delta less zero

Answer:
Example usage:

```sql
$increase(Requests) FROM requests
```

Query will be transformed into:

```sql
SELECT
    t,
    if(runningDifference(max_0) < 0, 0, runningDifference(max_0) ) AS max_0_Increase
FROM
(
    SELECT
        (intDiv(toUInt32(EventTime), 60) * 60) * 1000 AS t,
        max(Requests) AS max_0
    FROM requests
    WHERE ((EventDate >= toDate(1535711819)) AND (EventDate <= toDate(1535714715)))
    AND ((EventTime >= toDateTime(1535711819)) AND (EventTime <= toDateTime(1535714715)))
    GROUP BY t
    ORDER BY t
)
```

// see [issue 455](https://github.com/Altinity/clickhouse-grafana/issues/455) for the background

---


Question: How to: $increaseColumns(key, value) - is a combination of $columns and $increase for Counter-like metrics

Answer:
Example usage:

```sql
$increaseColumns(Protocol, Requests) FROM requests WHERE Protocol in ('udp','tcp')
```

Query will be transformed into:

```sql
SELECT
    t,
    groupArray((increaseColumns, max_0_Increase)) AS groupArr
FROM
(
    SELECT
        t,
        Protocol,
        if (runningDifference(max_0) < 0 OR neighbor(increaseColumns,-1,increaseColumns) != increaseColumns, 0, runningDifference(max_0)) AS max_0_Increase
    FROM
    (
        SELECT
            (intDiv(toUInt32(EventTime), 60) * 60) * 1000 AS t,
            Protocol AS increaseColumns,
            max(Requests) AS max_0
        FROM requests
        WHERE ((EventDate >= toDate(1535711819)) AND (EventDate <= toDate(1535714715)))
        AND ((EventTime >= toDateTime(1535711819)) AND (EventTime <= toDateTime(1535714715)))
        AND (Protocol IN ('udp', 'tcp'))
        GROUP BY
            t,
            Protocol
        ORDER BY
            t,
            Protocol
    )
)
GROUP BY t
ORDER BY t
```

// see [issue 455](https://github.com/Altinity/clickhouse-grafana/issues/455) for the background

---


Question: How to: Query Variable

Answer:
If you add a template variable of the type `Query`, you can write a ClickHouse query that can
return things like measurement names, key names or key values that are shown as a dropdown select box.

For example, you can have a variable that contains all values for the `hostname` column in a table if you specify a query like this in the templating variable *Query* setting.

```sql
SELECT hostname FROM host
```

To use time range dependent macros like `timeFilterByColumn($column)` in your query the refresh mode of the template variable needs to be set to *On Time Range Change*.

```sql
SELECT event_name FROM event_log WHERE $timeFilterByColumn(time_column)
```

Another option is a query that can create a key/value variable. The query should return two columns that are named `__text` and `__value`. The `__text` column value should be unique (if it is not unique then the first value will use). The options in the dropdown will have a text and value that allows you to have a friendly name as text and an id as the value. An example query with `hostname` as the text and `id` as the value:

```sql
SELECT hostname AS __text, id AS __value FROM host
```

You can also create nested variables. For example if you had another variable named `region`. Then you could have the hosts variable only show hosts from the current selected region with a query like this (if `region` is a multi-value variable then use the `IN` comparison operator rather than `=` to match against multiple values):

```sql
SELECT hostname FROM host WHERE region IN ($region)
```


Question: How to: Conditional Predicate

Answer:
If you are using templating to feed your predicate, you will face performance degradation when everything will select as the predicate, and it's not necessary. It's also true for textbox when nothing is entered, you have to write specific sql code to handle that.

To resolve this issue a new macro $conditionalTest(SQL Predicate,$variable) can be used to remove some part of the query.
If the variable is type query with all selected or if the variable is a textbox with nothing enter, then the SQL Predicate is not include in the generated query.

To give an example:
with 2 variables
  $var query with include All option
  $text textbox
  $text_with_single_quote textbox with single quote

  The following query

  ```sql
   SELECT
     $timeSeries as t,
     count()
     FROM $table
     WHERE $timeFilter
      $conditionalTest(AND toLowerCase(column) in ($var),$var)
      $conditionalTest(AND toLowerCase(column2) like '%$text%',$text)
      $conditionalTest(AND toLowerCase(column3) ilike ${text_with_single_quote:sqlstring},$text_with_single_quote)
     GROUP BY t
     ORDER BY t
  ```

   if the `$var` is selected as "All" value, and the `$text` variable is empty, the query will be converted into:

  ```sql
    SELECT
      $timeSeries as t,
      count()
       FROM $table
       WHERE $timeFilter
     GROUP BY t
     ORDER BY t
  ```

  If the `$var` template variable have select some elements, and the `$text` template variable has at least one char, the query will be converted into:

  ```sql
  SELECT
      $timeSeries as t,
      count()
       FROM $table
       WHERE $timeFilter
     AND toLowerCase(column) in ($var)
     AND toLowerCase(column2) like '%$text%'
     GROUP BY t
     ORDER BY t
 ```


Question: How to: Extended Conditional Test with Else Clause

Answer:
A new signature of the macro now supports three parameters:

```sql
$conditionalTest(SQL_if, SQL_else, $variable)
 ```

If the variable is type query with all selected or if the variable is a textbox with nothing entered, then the SQL_if is included in the generated query. Otherwise, the SQL_else is included.

To give an example:
with 2 variables
  $var query with include All option
  $text textbox
  $text_with_single_quote textbox with single quote

  The following query

  ```sql
   SELECT
     $timeSeries as t,
     count()
     FROM $table
     WHERE $timeFilter
      $conditionalTest(AND toLowerCase(column) in ($var), AND toLowerCase(column) in ($var), $var)
      $conditionalTest(AND toLowerCase(column2) like '%$text%', AND toLowerCase(column2) like '%$text%', $text)
      $conditionalTest(AND toLowerCase(column3) ilike ${text_with_single_quote:sqlstring}, AND toLowerCase(column3) ilike ${text_with_single_quote:sqlstring}, $text_with_single_quote)
     GROUP BY t
     ORDER BY t
  ```

   if the `$var` is selected as "All" value, and the `$text` variable is empty, the query will be converted into:

  ```sql
    SELECT
      $timeSeries as t,
      count()
       FROM $table
       WHERE $timeFilter
     GROUP BY t
     ORDER BY t
  ```

  If the `$var` template variable have select some elements, and the `$text` template variable has at least one char, the query will be converted into:

  ```sql
  SELECT
      $timeSeries as t,
      count()
       FROM $table
       WHERE $timeFilter
     AND toLowerCase(column) in ($var)
     AND toLowerCase(column2) like '%$text%'
     GROUP BY t
     ORDER BY t
 ```


Question: How to: Pie Chart ([https://grafana.com/plugins/grafana-piechart-panel](https://grafana.com/plugins/grafana-piechart-panel))

Answer:
Remember that pie chart plugin is not welcome for using in grafana - see [Grafana BLog - Friends don't let friends abuse pie charts](https://grafana.com/blog/2015/12/04/friends-dont-let-friends-abuse-pie-charts)

![top users](https://github.com/Altinity/clickhouse-grafana/raw/master/.github/images/09_requests_by_user_pie_chart.png)

To create "Top 5" diagram we will need two queries: one for 'Top 5' rows and one for 'Other' row.

Top5:

```sql
SELECT
    1 AS t, /* fake timestamp value */
    UserName,
    sum(Requests) AS Reqs
FROM requests
GROUP BY t, UserName
ORDER BY Reqs DESC
LIMIT 5
```

Other:

```sql
SELECT
    1 AS t, /* fake timestamp value */
    UserName,
    sum(Requests) AS Reqs
FROM requests
GROUP BY t, UserName
ORDER BY Reqs DESC
LIMIT 5,10000000000000 /* select some ridiculous number after first 5 */
```


Question: How to: Table view ([https://grafana.com/plugins/table](https://grafana.com/plugins/table))

Answer:
There are don't contain any tricks in displaying time-series data. To print summary data, omit time column, and format the result as "Table" and press "Run query".

```sql
SELECT
    UserName,
    sum(Requests) as Reqs
FROM requests
GROUP BY
    UserName
ORDER BY
    Reqs
```

![table view](https://github.com/Altinity/clickhouse-grafana/raw/master/.github/images/10_table_view.png)


Question: How to: Vertical histogram ([https://grafana.com/plugins/graph](https://grafana.com/plugins/graph))

Answer:
![vertical histogram](https://github.com/Altinity/clickhouse-grafana/raw/master/.github/images/11_vertical_histogram.png)

To make the vertical histogram from graph panel we will need to edit some settings:

* Display -> Draw Modes -> Bars
* Axes -> X-Axis -> Mode -> Series

You can use next query:

```sql
$columns(
    Size,
    sum(Items) Items)
FROM some_table
```

// It is also possible to use query without macros


Question: How to: Worldmap panel ([https://github.com/grafana/worldmap-panel](https://github.com/grafana/worldmap-panel))

Answer:
![worldmap](https://github.com/Altinity/clickhouse-grafana/raw/master/.github/images/12_worldmap_example.png)

If you have a table with country/city codes:

```sql
SELECT
    1,
    Country AS c,
    sum(Requests) AS Reqs
FROM requests
GLOBAL ANY INNER JOIN
(
    SELECT Country, CountryCode
    FROM countries
) USING (CountryCode)
WHERE $timeFilter
GROUP BY
    c
ORDER BY Reqs DESC
```

If you are using [geohash](https://github.com/grafana/worldmap-panel#geohashes-as-the-data-source) set following options:

![Format](https://github.com/Altinity/clickhouse-grafana/raw/master/.github/images/13_worldmap_format.png)

You can make following query with `Table` formatting:

![geohash-query](https://github.com/Altinity/clickhouse-grafana/raw/master/.github/images/14_worldmap_query.png)


Question: How to: Ad-hoc filters

Answer:
If there is an Ad-hoc variable, plugin will fetch all columns of all tables of all databases (except system database) as tags.
So in dropdown menu will be options like `database.table.column`. If you specify the default database it will only fetch tables and columns from that database, and the dropdown menu will have an option like `table.column`.
If there are ENUM columns, the plugin will fetch their options and use them as tag values.
Also, plugin will fetch 300 unique values for fields with other types.

Plugin will apply Ad-hoc filters to all queries on the dashboard if their settings `$database` and `$table` are the same
as `database.table` specified in Ad-hoc control. If the ad-hoc filter doesn't specify a table, it will apply to all queries regardless of the table.
This is useful if the dashboard contains queries to multiple different tables.

![ad-hoc](https://github.com/Altinity/clickhouse-grafana/raw/master/.github/images/15_adhoc_filter.png)

> There are no option to apply OR operator for multiple Ad-hoc filters - see grafana/grafana#10918
> There are no option to use IN operator for Ad-hoc filters due to Grafana limitations

There may be cases when CH contains too many tables and columns so their fetching could take notably amount of time. So, if you need
to have multiple dashboards with different databases using of `default database` won't help. The best way to solve this will be to have parametrized
ad-hoc variable in dashboard settings. Currently, it's not supported by Grafana interface (see [issue](https://github.com/grafana/grafana/issues/13109)).
As a temporary workaround, plugin will try to look for variable with name `adhoc_query_filter` and if it exists will use its value as query to fetch columns.
For this purpose we recommend creating some variable `constant` with the name `adhoc_query_filter` and set the value similar to the following one:

```sql
SELECT database, table, name, type FROM system.columns WHERE table='myTable' ORDER BY database, table
```

That should help to control data fetching by ad-hoc queries.


Question: How to: Template variable values via Query

Answer:
To use time range dependent macros like `$from` and `$to` in your query the refresh mode of the template variable needs to be set to On Time Range Change.

```sql
SELECT ClientID FROM events WHERE EventTime > toDateTime($from) AND EventTime < toDateTime($to)
```


Question: How to: Annotations

Answer:
Plugin support Annotations with regions. To enable this feature open Dashboard `settings` and add new annotation query with `clickhouse` datasource with properly field names.

![Annotation query add](https://github.com/Altinity/clickhouse-grafana/raw/master/.github/images/16_annotations_query_add.png)

![Annotation query example](https://github.com/Altinity/clickhouse-grafana/raw/master/.github/images/17_annotations_query_example.png)

![Annotation with regions graph panel](https://github.com/Altinity/clickhouse-grafana/raw/master/.github/images/18_annotations_graph.png)


Question: How to: Alerts support

Answer:
Grafana provide two kind of alerts. Unified alerts and graph panel related alerts (legacy).
Both kind of alerts supports by our plugin can't be used together.
Use `GF_UNIFIED_ALERTING_ENABLED=1` (preferable) or `GF_ALERTING_ENABLED=1` environment variables for switch.


Question: How to: Panel related alerts (legacy)

Answer:
To enable alerts open "alerts" tab in panel, and define alert expression as described on [grafana.com](https://grafana.com/docs/grafana/latest/alerting/)

![Alerts in graph panel](https://github.com/Altinity/clickhouse-grafana/raw/master/.github/images/19_alerts_tab.png)

Be careful with Template variables values, currently grafana doesn't support template variables in alert queries itself.
Also, grafana UI doesn't pass template variables values to a backend, after you change it on frontend UI.

So, the clickhouse grafana plugin can use template variables values, because we have "Generated SQL" which pass to backend "as is"
To ensure template variables values will properly pass to a backend part of the plugin.
Please choose the required template variables values for your alerts in UI dropdown,
ensure values properly rendered in "Generated SQL" (maybe need change SQL queries in query editor)
and save a whole dashboard to the Grafana server

WARNING: `Test alert` button doesn't save a current state of alert rules to a backend part of the plugin.

If the "Generated SQL" properly passed into backend part of plugin, you will see something like this:
![Graph panel with alerts](https://github.com/Altinity/clickhouse-grafana/raw/master/.github/images/20_alerts_panel.png)


Question: How to: Unified Alerts support

Answer:
Unified alerts could be provisioned with YAML file, look to https://github.com/Altinity/clickhouse-grafana/tree/master/docker/grafana/provisioning/alerting/

![Unified alerts menu](https://github.com/Altinity/clickhouse-grafana/raw/master/.github/images/21_unified_alerts_menu.png)

![Unified alerts panel](https://github.com/Altinity/clickhouse-grafana/raw/master/.github/images/22_unified_alerts_adding.png)

To export exists unified alerts to YAML use Export alerts

![Unified alerts export](https://github.com/Altinity/clickhouse-grafana/raw/master/.github/images/24_alerts_export.png)


Question: How to: Alerts troubleshooting

Answer: To troubleshoot alerts in clickhouse grafana plugin when enable `level=debug` in `log` section `grafana.ini` or via `GF_LOG_LEVEL=debug` environment variable.


Question: How to: Histogram support

Answer:
![Histogram](https://github.com/Altinity/clickhouse-grafana/raw/master/src/img/histogram.png)

To show Histogram you need query in format as "Time Series"

According to https://grafana.com/docs/grafana/latest/panels-visualizations/visualizations/histogram, Histograms support time series and any table results with one or more numerical fields.


Question: How to: Logs support

Answer:
To render your ClickHouse data as Logs, please use special format in "Format as" dropdown in Query Editor called "Logs". This option helps Grafana recognizes data as logs and shows logs visualization automatically in Explore UI. On dashboards you can use [Logs panel](https://grafana.com/docs/grafana/latest/visualizations/logs-panel/) as well.

![Format as Logs](https://github.com/Altinity/clickhouse-grafana/raw/master/.github/images/23_logs_support.png)

To return suitable for logs data - query should return at least one time field (assumed that it will be first field) and one text field from the ClickHouse.

Plugin is also transforming all text fields, except log line, into the labels using following rules:

* Log line will be taken either from dedicated `content` field or from first in order text field in result
* All other text fields will be treated as a labels

There are few dedicated fields that are recognized by Grafana:

* `level` (string) - set the level for each log line
* `id` (string) - by default, Grafana offers basic support for deduplicating log lines, that can be improved by adding this field to explicitly assign identifiers to each log line

All other fields returned from data source will be recognized by Grafana as [detected fields](https://grafana.com/docs/grafana/latest/explore/logs-integration/#labels-and-detected-fields)


Question: How to: Flamegraph support

Answer:
![Format as: Flamegraph](https://github.com/Altinity/clickhouse-grafana/raw/master/.github/images/25_format_as_flamegraph.png)

To show Flamegraph you need query in format as "Flame Graph"
According to https://grafana.com/docs/grafana/latest/panels-visualizations/visualizations/flame-graph/#data-api, you need to have recordset with 4 fields
- `level` - Numeric - the level of the stack frame. The root frame is level 0.
- `label` - String - the function name or other symbol which identify
- `value` - Numeric - the number of samples or bytes that were recorded in this stack trace
- `self` - Numeric - the number of samples or bytes that were recorded in only this stack frame excluding the children, for clickhouse this is usually zero, but for the last frame in stack requires `self` equals with `value` to properly flamegraph vizualization

**Moreover, rows shall be ordered by stack trace and level**

If you setup `query_profiler_real_time_period_ns` in profile or query level settings when you can try to visualize it as FlameGraph with the following query
Look to [system.trace_log](https://clickhouse.com/docs/en/operations/system-tables/trace_log) table description for how to get data for FlameGraph
Look to [flamegraph dashboard example](https://github.com/Altinity/clickhouse-grafana/blob/master/docker/grafana/dashboards/flamegraph_and_tracing_support.json) for example of dashboard with FlameGraph


Question: How to: Flamegraph query example:

Answer:
```sql
SELECT length(trace)  - level_num AS level, label, count() AS value, 0 self
FROM system.trace_log
  ARRAY JOIN arrayEnumerate(trace) AS level_num,
  arrayMap(x -> if(addressToSymbol(x) != '', demangle(addressToSymbol(x)), 'unknown') , trace) AS label
WHERE trace_type='Real' AND $timeFilter
GROUP BY level, label, trace
ORDER BY trace, level
```


Question: How to: Traces support

Answer:
To show Traces you need query with format as "Traces" with following
![Format as Traces](https://github.com/Altinity/clickhouse-grafana/raw/master/.github/images/26_format_as_trace.png)

![Trace example](https://github.com/Altinity/clickhouse-grafana/raw/master/.github/images/27_traces_example.png)

For example, if `
1
` in user profile and `system.opentelemetry_span_log` is not emtpy, then you can show traces about clickhouse query execution
Look to [system.opentelemetry_span_log](https://clickhouse.com/docs/en/operations/system-tables/opentelemetry_span_log) table description for how to get data for FlameGraph
Look to [tracing dashboard example](https://github.com/Altinity/clickhouse-grafana/blob/master/docker/grafana/dashboards/flamegraph_and_tracing_support.json) for example of dashboard with FlameGraph

Tracing visualization requires following field names (case sensitive):
- `traceID` - String
- `spanID` - String
- `operationName` - String
- `parentSpanID` - String
- `serviceName` - String
- `duration` - UInt64 - duration in milliseconds
- `startTime` - UInt64 - start time in milliseconds
- `tags` - map(String, String) - tags for span
- `serviceTags` - map(String, String) - tags for service (for example 'hostName')


Question: How to: Traces query example for system.opentelemetry_span_log

Answer:
```sql
SELECT
  trace_id AS traceID,
  span_id AS spanID,
  operation_name AS operationName,
  parent_span_id AS parentSpanID,
  'clickhouse' AS serviceName,
  intDiv(finish_time_us - start_time_us, 1000) AS duration,
  intDiv(start_time_us,1000) AS startTime,
  attribute AS tags,
  map('hostName',hostname) AS serviceTags
FROM
  system.opentelemetry_span_log
WHERE $timeFilter
ORDER BY traceID, startTime
```


Question: How to: Configure the Datasource with Provisioning

Answer:
It’s now possible to configure datasources using config files with Grafana’s provisioning system.
You can read more about how it works and all the settings you can set for datasources on the [provisioning docs page](http://docs.grafana.org/administration/provisioning/#datasources).

Here are some provisioning example:

```yaml
apiVersion: 1

datasources:
 - name: Clickhouse
   type: vertamedia-clickhouse-datasource
   access: proxy
   url: http://localhost:8123
   #
enable/disable basic auth
   basicAuth: false
   #
basic auth username
   basicAuthUser: "default"
   #
enable/disable with credentials headers
   withCredentials: false
   #
mark as default datasource. Max one per org
   isDefault: false
   #
fields that will be converted to json and stored in json_data
   jsonData:
     #
enable/disable sending 'add_http_cors_header=1' parameter
     addCorsHeader: false
     #
enable/disable using POST method for sending queries
     usePOST: false
     #
enable/disable using Accept-Encoding header in each request
     useCompression: false
     #
compression type allowed values: gzip, zstd, br, deflate
     compressionType: ""
     #
default database name
     defaultDatabase: ""
     #
enable/disable tls authorization
     tlsAuth: false
     #
enable/disable tls authorization with custom ca
     tlsAuthWithCACert: false
     #
enable/disable authorization with X-ClickHouse-* headers
     useYandexCloudAuthorization: false
     #
X-ClickHouse-Key header value for authorization
     xHeaderUser: ""
     #
the same value as url when `useYandexCloudAuthorization: true`
     # @todo remove this workaround when merge https://github.com/grafana/grafana/pull/80858
     dataSourceUrl: "http://localhost:8123"
   secureJsonData:
     #
X-ClickHouse-User header value for authorization
     xHeaderKey: ""
     #
basic auth password
     basicAuthPassword: ""
     #
custom certificate authority for TLS https connection, base64 encoded
     tlsCACert: ""
     #
custom client certificate for TLS https connection, base64 encoded
     tlsClientCert: ""
     #
custom client secret key for TLS https connection, base64 encoded
     tlsClientKey: ""
```

Some settings and security params are the same for all datasources. You can find them [here](http://docs.grafana.org/administration/provisioning/#example-datasource-config-file).


Question: How to: Contributing

Answer:
If you have any idea for an improvement or found a bug do not hesitate to open an issue or submit a pull request.
We will appreciate any help from the community which will make working with such amazing products as ClickHouse and Grafana more convenient.


Question: How to: Development

Answer:
see [CONTRIBUTING.md](https://github.com/Altinity/clickhouse-grafana/blob/master/CONTRIBUTING.md) for Development and Pull request Contributing instructions

License

---
MIT License, please see [LICENSE](https://github.com/Altinity/clickhouse-grafana/blob/master/LICENSE) for details.


Question: How to: Introduction

Answer:
__How They SRE__ How They SRE is a curated knowledge repository of Site Reliability Engineering (SRE) best practices, tools, techniques, and culture adopted by leading technology or tech-savvy organizations.

Numerous organizations frequently share their insights and expertise, encompassing best practices, tools, and techniques that shape their engineering culture. They do this through various public platforms such as engineering blogs, conferences, and meetups. This repository compiles and presents content gathered from these sources.


Question: How to: Blog Posts

Answer:
* [Enter the Abattoir - Building 'à la carte' gitops tooling](https://achievers.engineering/enter-the-abattoir-ee5e2019f0b3)
* [Scaling Production Globally — The service mesh facelift (Part-1)](https://achievers.engineering/scaling-production-globally-service-mesh-face-lift-part-1-30ad6d393d04)
* [Scaling Production Globally - Solving observability problems for developers (Part-2)](https://achievers.engineering/scaling-production-globally-solving-observability-problems-for-developers-part-2-b5416ce5eb8a)
* [Load Testing Kubernetes: Building a Framework (Part-1)](https://achievers.engineering/load-testing-kubernetes-building-a-framework-part-1-bdc0af4ae7e2)
* [Load Testing Kubernetes: Resolving bottlenecks and improving performance (Part-2)](https://achievers.engineering/load-testing-kubernetes-resolving-bottlenecks-and-improving-performance-part-2-c4f08102f105)
Airbnb


Question: How to: Major incidents & analysis reports

Answer:
* [Information on the Capital One Cyber Incident](https://www.capitalone.com/facts2019/)
* [A Case Study of the Capital One Data Breach](http://web.mit.edu/smadnick/www/wp/2020-16.pdf)


Question: How to: Other Resources

Answer:
#### Awesome Lists

* [Awesome SRE](https://github.com/dastergon/awesome-sre)
* [Awesome Site Reliability Engineering Tools](https://github.com/SquadcastHub/awesome-sre-tools)
* [Awesome Chaos Engineering](https://github.com/dastergon/awesome-chaos-engineering)
* [Awesome Monitoring](https://github.com/crazy-canux/awesome-monitoring)
* [Awesome Observability](https://github.com/adriannovegil/awesome-observability)
* [Awesome MLOps](https://github.com/visenger/awesome-mlops)
* [ML-Ops.org](https://ml-ops.org/)

#### SRE Resources from various organizations

* [Google SRE Page](https://sre.google/)
* [Google SRE Classroom](https://sre.google/classroom/)
* [Google Cloud SRE Page](https://cloud.google.com/sre)
* [Microsoft SRE Page](https://docs.microsoft.com/en-us/azure/site-reliability-engineering/)
* [School of SRE from LinkedIn](https://linkedin.github.io/school-of-sre/)
* [Stripe Increment Magazine Issue 16 on Reliability](https://increment.com/reliability/)
* [AWS Observability Recipes](https://aws-observability.github.io/aws-o11y-recipes/)
* [Awesome Sysadmin](https://github.com/awesome-foss/awesome-sysadmin)

#### Incidents & postmortems

* [The Verica Open Incident Database](https://www.thevoid.community/)
* [Postmortem Templates](https://github.com/dastergon/postmortem-templates)
* [Incident Review and Postmortem Best Practices](https://blog.pragmaticengineer.com/postmortem-best-practices/)

#### Newsletters

* [SRE Weekly Newsletter](https://sreweekly.com/)
* [Chaos Engineering Newsletter](https://chaosengineering.news/)
* [DevOps Weekly Newsletter](http://devopsweekly.com)


Question: How to: Other How They... repos

Answer:
* [Howtheytest](https://github.com/abhivaikar/howtheytest)
* [Howtheydevops](https://github.com/bregman-arie/howtheydevops)
* [Howtheyaws](https://github.com/upgundecha/howtheyaws)
* [Applied AI](https://github.com/upgundecha/applied-ai)


Question: How to: Table of Contents

Answer:
- [Table of Contents](#table-of-contents)
- [Why Gatus?](#why-gatus)
- [Features](#features)
- [Usage](#usage)
- [Configuration](#configuration)
  - [Endpoints](#endpoints)
  - [External Endpoints](#external-endpoints)
  - [Suites (ALPHA)](#suites-alpha)
  - [Conditions](#conditions)
    - [Placeholders](#placeholders)
    - [Functions](#functions)
  - [Web](#web)
  - [UI](#ui)
  - [Announcements](#announcements)
  - [Storage](#storage)
  - [Client configuration](#client-configuration)
  - [Tunneling](#tunneling)
  - [Alerting](#alerting)
    - [Configuring AWS SES alerts](#configuring-aws-ses-alerts)
    - [Configuring ClickUp alerts](#configuring-clickup-alerts)
    - [Configuring Datadog alerts](#configuring-datadog-alerts)
    - [Configuring Discord alerts](#configuring-discord-alerts)
    - [Configuring Email alerts](#configuring-email-alerts)
    - [Configuring Gitea alerts](#configuring-gitea-alerts)
    - [Configuring GitHub alerts](#configuring-github-alerts)
    - [Configuring GitLab alerts](#configuring-gitlab-alerts)
    - [Configuring Google Chat alerts](#configuring-google-chat-alerts)
    - [Configuring Gotify alerts](#configuring-gotify-alerts)
    - [Configuring HomeAssistant alerts](#configuring-homeassistant-alerts)
    - [Configuring IFTTT alerts](#configuring-ifttt-alerts)
    - [Configuring Ilert alerts](#configuring-ilert-alerts)
    - [Configuring Incident.io alerts](#configuring-incidentio-alerts)
    - [Configuring Line alerts](#configuring-line-alerts)
    - [Configuring Matrix alerts](#configuring-matrix-alerts)
    - [Configuring Mattermost alerts](#configuring-mattermost-alerts)
    - [Configuring Messagebird alerts](#configuring-messagebird-alerts)
    - [Configuring n8n alerts](#configuring-n8n-alerts)
    - [Configuring New Relic alerts](#configuring-new-relic-alerts)
    - [Configuring Ntfy alerts](#configuring-ntfy-alerts)
    - [Configuring Opsgenie alerts](#configuring-opsgenie-alerts)
    - [Configuring PagerDuty alerts](#configuring-pagerduty-alerts)
    - [Configuring Plivo alerts](#configuring-plivo-alerts)
    - [Configuring Pushover alerts](#configuring-pushover-alerts)
    - [Configuring Rocket.Chat alerts](#configuring-rocketchat-alerts)
    - [Configuring SendGrid alerts](#configuring-sendgrid-alerts)
    - [Configuring Signal alerts](#configuring-signal-alerts)
    - [Configuring SIGNL4 alerts](#configuring-signl4-alerts)
    - [Configuring Slack alerts](#configuring-slack-alerts)
    - [Configuring Splunk alerts](#configuring-splunk-alerts)
    - [Configuring Squadcast alerts](#configuring-squadcast-alerts)
    - [Configuring Teams alerts *(Deprecated)*](#configuring-teams-alerts-deprecated)
    - [Configuring Teams Workflow alerts](#configuring-teams-workflow-alerts)
    - [Configuring Telegram alerts](#configuring-telegram-alerts)
    - [Configuring Twilio alerts](#configuring-twilio-alerts)
    - [Configuring Vonage alerts](#configuring-vonage-alerts)
    - [Configuring Webex alerts](#configuring-webex-alerts)
    - [Configuring Zapier alerts](#configuring-zapier-alerts)
    - [Configuring Zulip alerts](#configuring-zulip-alerts)
    - [Configuring custom alerts](#configuring-custom-alerts)
    - [Setting a default alert](#setting-a-default-alert)
  - [Maintenance](#maintenance)
  - [Security](#security)
    - [Basic Authentication](#basic-authentication)
    - [OIDC](#oidc)
  - [TLS Encryption](#tls-encryption)
  - [Metrics](#metrics)
    - [Custom Labels](#custom-labels)
  - [Connectivity](#connectivity)
  - [Remote instances (EXPERIMENTAL)](#remote-instances-experimental)
- [Deployment](#deployment)
  - [Docker](#docker)
  - [Helm Chart](#helm-chart)
  - [Terraform](#terraform)
    - [Kubernetes](#kubernetes)
- [Running the tests](#running-the-tests)
- [Using in Production](#using-in-production)
- [FAQ](#faq)
  - [Sending a GraphQL request](#sending-a-graphql-request)
  - [Recommended interval](#recommended-interval)
  - [Default timeouts](#default-timeouts)
  - [Monitoring a TCP endpoint](#monitoring-a-tcp-endpoint)
  - [Monitoring a UDP endpoint](#monitoring-a-udp-endpoint)
  - [Monitoring a SCTP endpoint](#monitoring-a-sctp-endpoint)
  - [Monitoring a WebSocket endpoint](#monitoring-a-websocket-endpoint)
  - [Monitoring an endpoint using gRPC](#monitoring-an-endpoint-using-grpc)
  - [Monitoring an endpoint using ICMP](#monitoring-an-endpoint-using-icmp)
  - [Monitoring an endpoint using DNS queries](#monitoring-an-endpoint-using-dns-queries)
  - [Monitoring an endpoint using SSH](#monitoring-an-endpoint-using-ssh)
  - [Monitoring an endpoint using STARTTLS](#monitoring-an-endpoint-using-starttls)
  - [Monitoring an endpoint using TLS](#monitoring-an-endpoint-using-tls)
  - [Monitoring domain expiration](#monitoring-domain-expiration)
  - [Concurrency](#concurrency)
  - [Reloading configuration on the fly](#reloading-configuration-on-the-fly)
  - [Endpoint groups](#endpoint-groups)
  - [How do I sort by group by default?](#how-do-i-sort-by-group-by-def


Question: How to: Why Gatus?

Answer:
Before getting into the specifics, I want to address the most common question:
> Why would I use Gatus when I can just use Prometheus’ Alertmanager, Cloudwatch or even Splunk?

Neither of these can tell you that there’s a problem if there are no clients actively calling the endpoint.
In other words, it's because monitoring metrics mostly rely on existing traffic, which effectively means that unless
your clients are already experiencing a problem, you won't be notified.

Gatus, on the other hand, allows you to configure health checks for each of your features, which in turn allows it to
monitor these features and potentially alert you before any clients are impacted.

A sign you may want to look into Gatus is by simply asking yourself whether you'd receive an alert if your load balancer
was to go down right now. Will any of your existing alerts be triggered? Your metrics won’t report an increase in errors
if no traffic makes it to your applications. This puts you in a situation where your clients are the ones
that will notify you about the degradation of your services rather than you reassuring them that you're working on
fixing the issue before they even know about it.


Question: How to: Configuration

Answer:
By default, the configuration file is expected to be at `config/config.yaml`.

You can specify a custom path by setting the `GATUS_CONFIG_PATH` environment variable.

If `GATUS_CONFIG_PATH` points to a directory, all `*.yaml` and `*.yml` files inside said directory and its
subdirectories are merged like so:
- All maps/objects are deep merged (i.e. you could define `alerting.slack` in one file and `alerting.pagerduty` in another file)
- All slices/arrays are appended (i.e. you can define `endpoints` in multiple files and each endpoint will be added to the final list of endpoints)
- Parameters with a primitive value (e.g. `metrics`, `alerting.slack.webhook-url`, etc.) may only be defined once to forcefully avoid any ambiguity
    - To clarify, this also means that you could not define `alerting.slack.webhook-url` in two files with different values. All files are merged into one before they are processed. This is by design.

> 💡 You can also use environment variables in the configuration file (e.g. `$DOMAIN`, `${DOMAIN}`)
>
> ⚠ When your configuration parameter contains a `$` symbol, you have to escape `$` with `$$`.
>
> See [Use environment variables in config files](#use-environment-variables-in-config-files) or [examples/docker-compose-postgres-storage/config/config.yaml](.examples/docker-compose-postgres-storage/config/config.yaml) for examples.

If you want to test it locally, see [Docker](#docker).