Question: How to: Scalable Infrastructure and performance

Answer:
* [The Authoritative Guide to Elasticsearch Performance Tuning (Part 1)](https://qbox.io/blog/authoritative-guide-elasticsearch-performance-tuning-part-1) [Part 2](https://qbox.io/blog/elasticsearch-performance-tuning-part-2-zen) [Part 3](https://qbox.io/index.php?p=blog/authoritative-guide-elasticsearch-performance-tuning-part-3)
* [Tuning data ingestion performance for Elasticsearch on Azure](https://azure.microsoft.com/en-us/documentation/articles/guidance-elasticsearch-tuning-data-ingestion-performance/) - and not only for Azure. That's a great article about Elasticsearch Performance testing by example
* [Elasticsearch Indexing Performance Cheatsheet](https://blog.codecentric.de/en/2014/05/elasticsearch-indexing-performance-cheatsheet/) - when you plan to index large amounts of data in Elasticsearch (by Patrick Peschlow)
* [Elasticsearch for Logging](http://edgeofsanity.net/article/2012/12/26/elasticsearch-for-logging.html) Elasticsearch configuration tips and tricks from Sanity
* [Scaling Elasticsearch to Hundreds of Developers](http://engineeringblog.yelp.com/2014/11/scaling-elasticsearch-to-hundreds-of-developers.html) by Joseph Lynch @yelp
* [10 Elasticsearch metrics to watch](http://radar.oreilly.com/2015/04/10-elasticsearch-metrics-to-watch.html)
* [Understanding Elasticsearch Performance](https://joshrendek.com/2015/11/understanding-elasticsearch-performance/)
* [Our Experience of Creating Large Scale Log Search System Using Elasticsearch](http://www.cubrid.org/blog/dev-platform/our-experience-creating-large-scale-log-search-system-using-elasticsearch/) - topology, separate master, data and search balancers nodes
* :open_file_folder: [Elasticsearch on Azure Guidance](https://github.com/Azure/azure-content/blob/master/articles/guidance/guidance-elasticsearch.md) it is 10% on Azure and 90% of a very valuable general information, tips and tricks about Elasticsearch
* [How to avoid the split-brain problem in Elasticsearch](http://blog.trifork.com/2013/10/24/how-to-avoid-the-split-brain-problem-in-elasticsearch/)
* Datadog's series about monitoring Elasticsearch performance:
  * [How to monitor Elasticsearch performance](https://www.datadoghq.com/blog/monitor-elasticsearch-performance-metrics/)
  * [How to collect Elasticsearch metrics](https://www.datadoghq.com/blog/collect-elasticsearch-metrics/)
  * [How to monitor Elasticsearch with Datadog](https://www.datadoghq.com/blog/monitor-elasticsearch-datadog/)
  * [How to solve 5 Elasticsearch performance and scaling problems](https://www.datadoghq.com/blog/elasticsearch-performance-scaling-problems/)
* [Performance Monitoring Essentials - Elasticsearch Edition](https://sematext.com/publications/performance-monitoring-essentials-elasticsearch-edition.pdf)
* [Operator for running Elasticsearch in Kubernetes](https://github.com/zalando-incubator/es-operator)


Question: How to: Integrations

Answer:
* [Apache Hive integration](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/hive.html)
* [Connecting Tableau to Elasticsearch (READ: How to query Elasticsearch with Hive SQL and Hadoop)](http://ryrobes.com/systems/connecting-tableau-to-elasticsearch-read-how-to-query-elasticsearch-with-hive-sql-and-hadoop/)
* [mradamlacey/elasticsearch-tableau-connector](https://github.com/mradamlacey/elasticsearch-tableau-connector)


Question: How to: Time series

Answer:
* [Elasticsearch as a Time Series Data Store](https://www.elastic.co/blog/elasticsearch-as-a-time-series-data-store) by Felix Barnsteiner
* [Running derivatives on Voyager velocity data](https://www.elastic.co/blog/out-of-this-world-aggregations) By Colin Goodheart-Smithe
* Shewhart Control Charts via Moving Averages: [Part 1](https://www.elastic.co/blog/staying-in-control-with-moving-averages-part-1) - [Part 2](https://www.elastic.co/blog/staying-in-control-with-moving-averages-part-2) by Zachary Tong
* Implementing a Statistical Anomaly Detector: [Part 1](https://www.elastic.co/blog/implementing-a-statistical-anomaly-detector-part-1) - [Part 2](https://www.elastic.co/blog/implementing-a-statistical-anomaly-detector-part-2) - [Part 3](https://www.elastic.co/blog/implementing-a-statistical-anomaly-detector-part-3) by Zachary Tong


Question: How to: Machine Learning

Answer:
* [Classifying images into Elasticsearch with DeepDetect](http://www.deepdetect.com/tutorials/es-image-classifier/) ([forum thread with discussion](https://discuss.elastic.co/t/categorizing-images-with-deep-learning-into-elasticsearch/33217)) by Emmanuel Benazera
* [Elasticsearch with Machine Learning](https://medium.com/hello-elasticsearch/elasticsearch-amazon-machine-learning-7d7b979c328d#.s50a6d5mn) ([English translation](https://translate.googleusercontent.com/translate_c?depth=1&hl=en&prev=search&rurl=translate.google.com&sl=ja&u=https://medium.com/hello-elasticsearch/elasticsearch-amazon-machine-learning-7d7b979c328d&usg=ALkJrhioEPGsVRglGPFTa6w2ZfM-ydSoeg)) by Kunihiko Kido
* [Recommender System with Mahout and Elasticsearch](https://www.mapr.com/products/mapr-sandbox-hadoop/tutorials/recommender-tutorial)


Question: How to: Use cases for Elasticsearch

Answer:
* [Data Infrastructure at IFTTT](https://medium.com/engineering-at-ifttt/data-infrastructure-at-ifttt-35414841f9b5) Elasticsearch, Kafka, Apache Spark, Redhsift, other AWS services
* [OFAC compliance with Elasticsearch](https://israelwebdev.wordpress.com/2015/01/19/ofac-compliance-with-elasticsearch/) using AWS
* [Building a Streaming Search Platform](https://blog.insightdatascience.com/building-a-streaming-search-platform-61a0d5a323a8#.f4b0rvae5) -
Streaming Search on Tweets: Storm, Elasticsearch, and Redis


Question: How to: Code, configuration file samples and other gists

Answer:
* [Elasticsearch config for a write-heavy cluster](https://gist.github.com/reyjrar/4364063) - reyjrar/elasticsearch.yml
* [chenryn/ESPL - Elastic Search Processing Language](https://github.com/chenryn/ESPL) PEG parser sample for SPL to Elasticsearch DSL
* [thomaspatzke/EQUEL](https://github.com/thomaspatzke/EQUEL) an Elasticsearch QUEry Language, based on G4 grammar parser


Question: How to: Who is using elasticsearch?


Question: How to: I want more! (Elasticsearch related resources)

Answer:
* [Technology Explained Blog](https://alexandreesl.com/)
* [EagerElk](http://blog.eagerelk.com/)
* [Tim Roes Blog](https://www.timroes.de/)


Question: How to: Contributing

Answer:
* Make sure you are about to post a valuable resource that belongs to this list
* Do NOT group ++Add and --Remove changes in same PR. Make them separate pull requests
* Use spellchecker
* All spelling and grammar corrections are welcome (except for the rule above)
* Fork this repo, do your edits, send the pull request
* Feel free to create any new sections
* Do not even try to add this repo to any awesome-awesome-* lists

#### ← [Awesome TypeScript](https://github.com/dzharii/awesome-typescript) -= Awesome Elasticsearch =-


Question: How to: Getting logspout

Answer:
Logspout is a very small Docker container (15.2MB virtual, based on [Alpine](https://github.com/gliderlabs/docker-alpine)). Pull the latest release from the index:

	$ docker pull gliderlabs/logspout:latest

You can also download and load a specific version:

	$ curl -s dl.gliderlabs.com/logspout/v2.tgz | docker load


Question: How to: Using logspout

Answer:
#### Route all container output to remote syslog

The simplest way to use logspout is to just take all logs and ship to a remote syslog. Just pass a syslog URI (or several comma separated URIs) as the command. Here we show use of the `tls` encrypted transport option in the URI. Also, we always mount the Docker Unix socket with `-v` to `/var/run/docker.sock`:

	$ docker run --name="logspout" \
		--volume=/var/run/docker.sock:/var/run/docker.sock \
		gliderlabs/logspout \
		syslog+tls://logs.papertrailapp.com:55555

logspout will gather logs from other containers that are started **without the `-t` option** and are configured with a logging driver that works with `docker logs` (`journald` and `json-file`).

To see what data is used for syslog messages, see the [syslog adapter](http://github.com/gliderlabs/logspout/blob/master/adapters) docs.

The container must be able to access the Docker Unix socket to mount it. This is typically a problem when [namespace remapping](https://docs.docker.com/engine/security/userns-remap/) is enabled. To disable remapping for the logspout container, pass the `--userns=host` flag to `docker run`, `.. create`, etc.

#### Ignoring specific containers

You can tell logspout to ignore specific containers by setting an environment variable when starting your container, like so:-

    $ docker run -d -e 'LOGSPOUT=ignore' image

Or, by adding a label which you define by setting an environment variable when running logspout:

    $ docker run --name="logspout" \
        -e EXCLUDE_LABEL=logspout.exclude \
        --volume=/var/run/docker.sock:/var/run/docker.sock \
        gliderlabs/logspout
    $ docker run -d --label logspout.exclude=true image

Logspout also allows to ignore containers by specifying a list of labels using the environment variables `EXCLUDE_LABELS` or `EXCLUDE_LABEL`, using the `;` as separator:

    $ $ docker run --name="logspout" \
        -e EXCLUDE_LABELS=k8s:app;backend:rails;io.kubernetes.pod.namespace:default \
        --volume=/var/run/docker.sock:/var/run/docker.sock \
        gliderlabs/logspout
    $ docker run -d --label k8s=app image1
    $ docker run -d --label backend=rails image2

**NOTE** Setting `EXCLUDE_LABELS` would take precedence over setting `EXCLUDE_LABEL`

#### Including specific containers

You can tell logspout to only include certain containers by setting filter parameters on the URI:

	$ docker run \
		--volume=/var/run/docker.sock:/var/run/docker.sock \
		gliderlabs/logspout \
		raw://192.168.10.10:5000?filter.name=*_db

	$ docker run \
		--volume=/var/run/docker.sock:/var/run/docker.sock \
		gliderlabs/logspout \
		raw://192.168.10.10:5000?filter.id=3b6ba57db54a

	$ docker run \
		--volume=/var/run/docker.sock:/var/run/docker.sock \
		gliderlabs/logspout \
		raw://192.168.10.10:5000?filter.sources=stdout%2Cstderr

	# Forward logs from containers with both label 'a' starting with 'x', and label 'b' ending in 'y'.
	$ docker run \
		--volume=/var/run/docker.sock:/var/run/docker.sock \
		gliderlabs/logspout \
		raw://192.168.10.10:5000?filter.labels=a:x*%2Cb:*y

Note that you must URL-encode parameter values such as the comma in `filter.sources` and `filter.labels`.

#### Multiple logging destinations

You can route to multiple destinations by comma-separating the URIs:

	$ docker run \
		--volume=/var/run/docker.sock:/var/run/docker.sock \
		gliderlabs/logspout \
		raw://192.168.10.10:5000?filter.name=*_db,syslog+tls://logs.papertrailapp.com:55555?filter.name=*_app

#### Suppressing backlog tail
You can tell logspout to only display log entries since container "start" or "restart" event by setting a `BACKLOG=false` environment variable (equivalent to `docker logs --since=0s`):

	$ docker run -d --name="logspout" \
		-e 'BACKLOG=false' \
		--volume=/var/run/docker.sock:/var/run/docker.sock \
		gliderlabs/logspout

The default behaviour is to output all logs since creation of the container (equivalent to `docker logs --tail=all` or simply `docker logs`).

> NOTE: Use of this option **may** cause the first few lines of log output to be missed following a container being started, if the container starts outputting logs before logspout has a chance to see them. If consistent capture of *every* line of logs is critical to your application, you might want to test thoroughly and/or avoid this option (at the expense of getting the entire backlog for every restarting container). This does not affect containers that are removed and recreated.

#### Environment variable, TAIL
Whilst BACKLOG=false restricts the tail by setting the Docker Logs.Options.Since to time.Now(), another mechanism to restrict the tail is to set TAIL=n.  Use of this mechanism avoids parsing the earlier content of the logfile which may have a speed advantage if the tail content is of no interest or has become corrupted.

#### Inspect log streams using curl

Using the [httpstream module](http://github.com/gliderlabs/logspout/blob/master/httpstream), you can connect with cur


Question: How to: TLS Settings

Answer:
logspout supports modification of the client TLS settings via environment variables described below:

| Environment Variable  | Description |
| :---                  |  :---       |
| `LOGSPOUT_TLS_DISABLE_SYSTEM_ROOTS` | when set to `true` it disables loading the system trust store into the trust store of logspout |
| `LOGSPOUT_TLS_CA_CERTS` | a comma separated list of filesystem paths to pem encoded CA certificates that should be added to logspout's TLS trust store. Each pem file can contain more than one certificate |
| `LOGSPOUT_TLS_CLIENT_CERT` | filesystem path to pem encoded x509 client certificate to load when TLS mutual authentication is desired |
| `LOGSPOUT_TLS_CLIENT_KEY` | filesystem path to pem encoded client private key to load when TLS mutual authentication is desired |
| `LOGSPOUT_TLS_HARDENING` | when set to `true` it enables stricter client TLS settings designed to mitigate some known TLS vulnerabilities |

#### Example TLS settings
The following settings cover some common use cases.
When running docker, use the `-e` flag to supply environment variables

**add your own CAs to the list of trusted authorities**
```
export LOGSPOUT_TLS_CA_CERTS="/opt/tls/ca/myRootCA1.pem,/opt/tls/ca/myRootCA2.pem"
```

**force logspout to ONLY trust your own CA**
```
export LOGSPOUT_TLS_DISABLE_SYSTEM_ROOTS=true
export LOGSPOUT_TLS_CA_CERTS="/opt/tls/ca/myRootCA1.pem"
```

**configure client authentication**
```
export LOGSPOUT_TLS_CLIENT_CERT="/opt/tls/client/myClient.pem"
export LOGSPOUT_TLS_CLIENT_KEY="/opt/tls/client/myClient-key.pem"
```

**highest possible security settings (paranoid mode)**
```
export LOGSPOUT_TLS_DISABLE_SYSTEM_ROOTS=true
export LOGSPOUT_TLS_HARDENING=true
export LOGSPOUT_TLS_CA_CERTS="/opt/tls/ca/myRootCA1.pem"
export LOGSPOUT_TLS_CLIENT_CERT="/opt/tls/client/myClient.pem"
export LOGSPOUT_TLS_CLIENT_KEY="/opt/tls/client/myClient-key.pem"
```


Question: How to: Builtin modules

Answer:
* adapters/raw
 * adapters/syslog
 * transports/tcp
 * transports/tls
 * transports/udp
 * httpstream
 * routesapi


Question: How to: Third-party modules

Answer:
* [logspout-kafka](https://github.com/dylanmei/logspout-kafka)
 * logspout-redis...
 * [logspout-logstash](https://github.com/looplab/logspout-logstash)
 * [logspout-redis-logstash](https://github.com/rtoma/logspout-redis-logstash)
 * [logspout-gelf](https://github.com/micahhausler/logspout-gelf) for Graylog
 * [logspout-fluentd](https://github.com/dsouzajude/logspout-fluentd) for fluentd or fluent-bit - instead of using fluentd log driver
 * [logspout-slack](https://github.com/kalisio/logspout-slack) for [Slack](https://slack.com/) notifications


Question: How to: Loggly support

Answer:
Use logspout to stream your docker logs to Loggly via the [Loggly syslog endpoint](https://www.loggly.com/docs/streaming-syslog-without-using-files/).
```
$ docker run --name logspout -d --volume=/var/run/docker.sock:/var/run/docker.sock \
    -e SYSLOG_STRUCTURED_DATA="
@41058 tag=\"some tag name\"" \
    gliderlabs/logspout \
    syslog+tcp://logs-01.loggly.com:514
```


Question: How to: Contributing

Answer:
As usual, pull requests are welcome. You can also propose releases by opening a PR against the `release` branch from `master`. Please be sure to bump the version and update `CHANGELOG.md` and include your changelog text in the PR body.

Discuss logspout development with us on Freenode in `#gliderlabs`.


Question: How to: Installation

Answer:
There are three common ways to run docker-gen:

- on the host
- bundled in a container with another application
- separate standalone containers

#### Host Install

Download the version you need, untar, and install to your PATH.

```console
wget https://github.com/nginx-proxy/docker-gen/releases/download/0.16.0/docker-gen-linux-amd64-0.16.0.tar.gz
tar xvzf docker-gen-linux-amd64-0.16.0.tar.gz
./docker-gen
```

#### Bundled Container Install

Docker-gen can be bundled inside of a container along-side applications.

[nginx-proxy/nginx-proxy](https://hub.docker.com/r/nginxproxy/nginx-proxy) trusted build is an example of
running docker-gen within a container along-side nginx.
[jwilder/docker-register](https://github.com/jwilder/docker-register) is an example of running
docker-gen within a container to do service registration with etcd.

#### Separate Container Install

It can also be run as two separate containers using the [nginx-proxy/docker-gen](https://hub.docker.com/r/nginxproxy/docker-gen)
image, together with virtually any other image.

This is how you could run the official [nginx](https://registry.hub.docker.com/_/nginx/) image and
have docker-gen generate a reverse proxy config in the same way that `nginx-proxy` works. You may want to do
this to prevent having the docker socket bound to a publicly exposed container service.

Start nginx with a shared volume:

```console
docker run -d -p 80:80 --name nginx -v /tmp/nginx:/etc/nginx/conf.d -t nginx
```

Fetch the template and start the docker-gen container with the shared volume:

```console
mkdir -p /tmp/templates && cd /tmp/templates
curl -o nginx.tmpl https://raw.githubusercontent.com/nginx-proxy/docker-gen/main/templates/nginx.tmpl
docker run -d --name nginx-gen --volumes-from nginx \
   -v /var/run/docker.sock:/tmp/docker.sock:rw \
   -v /tmp/templates:/etc/docker-gen/templates \
   -t nginxproxy/docker-gen -notify-sighup nginx -watch -only-exposed /etc/docker-gen/templates/nginx.tmpl /etc/nginx/conf.d/default.conf
```

Start a container, taking note of any Environment variables a container expects. See the top of a template for details.

```console
docker run --env VIRTUAL_HOST='example.com' --env VIRTUAL_PORT=80 ...
```

---


Question: How to: Configuration file

Answer:
Using the -config flag from above you can tell docker-gen to use the specified config file instead of command-line options. Multiple templates can be defined and they will be executed in the order that they appear in the config file.

An example configuration file, **docker-gen.cfg** can be found in the examples folder.

#### Configuration File Syntax

```ini
[[config]]


Question: How to: or the container id can be used followed by the signal to send

Answer:
```

Putting it all together here is an example configuration file.

```ini
[[config]]
template = "/etc/nginx/nginx.conf.tmpl"
dest = "/etc/nginx/sites-available/default"
onlyexposed = true
notifycmd = "/etc/init.d/nginx reload"

[[config]]
template = "/etc/logrotate.conf.tmpl"
dest = "/etc/logrotate.d/docker"
watch = true

[[config]]
template = "/etc/docker-gen/templates/nginx.tmpl"
dest = "/etc/nginx/conf.d/default.conf"
watch = true
wait = "500ms:2s"

[config.NotifyContainers]
nginx = 1  # 1 is a signal number to be sent; here SIGHUP
e75a60548dc9 = 1  # a key can be either container name (nginx) or ID
```

---


Question: How to: Templating

Answer:
The templates used by docker-gen are written using the Go [text/template](http://golang.org/pkg/text/template/) language. In addition to the [built-in functions](http://golang.org/pkg/text/template/#hdr-Functions) supplied by Go, docker-gen uses [sprig](https://masterminds.github.io/sprig/) and some additional functions to make it simpler (or possible) to generate your desired output. Some templates rely on environment variables within the container to make decisions on what to generate from the template.

Several templates may be parsed at once by using a semicolon (`;`) to delimit the `template` value. This can be used as a proxy for Golang's nested template functionality. In all cases, the main rendered template should go first.

```
[[config]]
template = "/etc/docker-gen/templates/nginx.tmpl;/etc/docker-gen/templates/header.tmpl"
dest = "/etc/nginx/conf.d/default.conf"
watch = true
wait = "500ms:2s"
```

#### Emit Structure

Within the templates, the object emitted by docker-gen will be a structure consisting of following Go structs:

```go
type RuntimeContainer struct {
    ID           string
    Created      time.Time
    Addresses    []Address
    Networks     []Network
    Gateway      string
    Name         string
    Hostname     string
    Image        DockerImage
    Env          map[string]string
    Volumes      map[string]Volume
    Node         SwarmNode
    Labels       map[string]string
    IP           string
    IP6LinkLocal string
    IP6Global    string
    Mounts       []Mount
    State        State
}

type Address struct {
    IP           string
    IP6LinkLocal string
    IP6Global    string
    Port         string
    HostPort     string
    Proto        string
    HostIP       string
}

type Network struct {
    IP                  string
    Name                string
    Gateway             string
    EndpointID          string
    IPv6Gateway         string
    GlobalIPv6Address   string
    MacAddress          string
    GlobalIPv6PrefixLen int
    IPPrefixLen         int
    Internal            bool
}

type DockerImage struct {
    Registry   string
    Repository string
    Tag        string
}

type Mount struct {
  Name        string
  Source      string
  Destination string
  Driver      string
  Mode        string
  RW          bool
}

type Volume struct {
    Path      string
    HostPath  string
    ReadWrite bool
}

type SwarmNode struct {
    ID      string
    Name    string
    Address Address
}

type State struct {
	Running bool
	Health  Health
}

type Health struct {
	Status string
}

// Accessible from the root in templates as .Docker
type Docker struct {
    Name                 string
    NumContainers        int
    NumImages            int
    Version              string
    ApiVersion           string
    GoVersion            string
    OperatingSystem      string
    Architecture         string
    CurrentContainerID   string
}

// Host environment variables accessible from root in templates as .Env
```

For example, this is a JSON version of an emitted RuntimeContainer struct:

```json
{
  "ID": "71e9768075836eb38557adcfc71a207386a0c597dbeda240cf905df79b18cebf",
  "Addresses": [
    {
      "IP": "172.17.0.4",
      "Port": "22",
      "Proto": "tcp",
      "HostIP": "192.168.10.24",
      "HostPort": "2222"
    }
  ],
  "Gateway": "172.17.42.1",
  "Node": {
    "ID": "I2VY:P7PF:TZD5:PGWB:QTI7:QDSP:C5UD:DYKR:XKKK:TRG2:M2BL:DFUN",
    "Name": "docker-test",
    "Address": {
      "IP": "192.168.10.24"
    }
  },
  "Labels": {
    "operatingsystem": "Ubuntu 14.04.2 LTS",
    "storagedriver": "devicemapper",
    "anything_foo": "something_bar"
  },
  "IP": "172.17.0.4",
  "Name": "docker_register",
  "Hostname": "71e976807583",
  "Image": {
    "Registry": "jwilder",
    "Repository": "docker-register"
  },
  "Env": {
    "ETCD_HOST": "172.17.42.1:4001",
    "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
    "DOCKER_HOST": "unix:///var/run/docker.sock",
    "HOST_IP": "172.17.42.1"
  },
  "Volumes": {
    "/mnt": {
      "Path": "/mnt",
      "HostPath": "/Users/joebob/tmp",
      "ReadWrite": true
    }
  }
}
```

#### Functions

- [Functions from Go](https://pkg.go.dev/text/template#hdr-Functions)
- [Functions from Sprig v3](https://masterminds.github.io/sprig/), except for those that have the same name as one of the following functions.
- _`closest $array $value`_: Returns the longest matching substring in `$array` that matches `$value`
- _`coalesce ...`_: Returns the first non-nil argument.
- _`comment $delimiter $string`_: Returns `$string` with each line prefixed by `$delimiter` (helpful for debugging combined with Sprig `toPrettyJson`: `{{ toPrettyJson $ | comment "#" }}`).
- _`contains $map $key`_: Returns `true` if `$map` contains `$key`. Takes maps from `string` to any type.
- _`dir $path`_: Returns an array of filenames in the specified `$path`.
- _`exists $path`_: Returns `true` if `$path` refers to an existing file or


Question: How to: Development

Answer:
This project uses [Go Modules](https://golang.org/ref/mod) for managing 3rd party dependencies.
This means that at least `go 1.11` is required.

For `go 1.11` and `go 1.12` it is additionally required to manually enable support by setting `GO111MODULE=on`.
For later versions, this is not required.

```console
git clone
cd
make get-deps
make
```


Question: How to: Powered by

Answer: [![GoLand logo](https://resources.jetbrains.com/storage/products/company/brand/logos/GoLand_icon.svg)](https://www.jetbrains.com/go/)


Question: How to: Installation

Answer:
Add this line to your application’s Gemfile:

```ruby
gem "ahoy_matey"
```

And run:

```sh
bundle install
rails generate ahoy:install
rails db:migrate
```

Restart your web server, open a page in your browser, and a visit will be created :tada:

Track your first event from a controller with:

```ruby
ahoy.track "My first event", language: "Ruby"
```


Question: How to: JavaScript, Native Apps, & AMP

Answer:
Enable the API in `config/initializers/ahoy.rb`:

```ruby
Ahoy.api = true
```

And restart your web server.


Question: How to: JavaScript

Answer:
For Importmap (Rails 7+ default), add to `config/importmap.rb`:

```ruby
pin "ahoy", to: "ahoy.js"
```

And add to `app/javascript/application.js`:

```javascript
import "ahoy"
```

For Bun, esbuild, rollup.js, or Webpack, run:

```sh
bun add ahoy.js


Question: How to: Native Apps

Answer: Check out [Ahoy iOS](https://github.com/namolnad/ahoy-ios) and [Ahoy Android](https://github.com/instacart/ahoy-android).


Question: How to: Geocoding Setup

Answer: To enable geocoding, see the [Geocoding section](#geocoding).


Question: How to: GDPR Compliance

Answer: Ahoy provides a number of options to help with GDPR compliance. See the [GDPR section](#gdpr-compliance-1) for more info.


Question: How to: Associated Models

Answer:
Say we want to associate orders with visits. Just add `visitable` to the model.

```ruby
class Order < ApplicationRecord
  visitable :ahoy_visit
end
```

When a visitor places an order, the `ahoy_visit_id` column is automatically set :tada:

See where orders are coming from with simple joins:

```ruby
Order.joins(:ahoy_visit).group("referring_domain").count
Order.joins(:ahoy_visit).group("city").count
Order.joins(:ahoy_visit).group("device_type").count
```

Here’s what the migration to add the `ahoy_visit_id` column should look like:

```ruby
class AddAhoyVisitToOrders < ActiveRecord::Migration[8.1]
  def change
    add_reference :orders, :ahoy_visit
  end
end
```

Customize the column with:

```ruby
visitable :sign_up_visit
```


Question: How to: Exclusions

Answer:
Bots are excluded from tracking by default. To include them, use:

```ruby
Ahoy.track_bots = true
```

Add your own rules with:

```ruby
Ahoy.exclude_method = lambda do |controller, request|
  request.ip == "192.168.1.1"
end
```


Question: How to: Visit Duration

Answer:
By default, a new visit is created after 4 hours of inactivity. Change this with:

```ruby
Ahoy.visit_duration = 30.minutes
```


Question: How to: Visitor Duration

Answer:
By default, a new `visitor_token` is generated after 2 years. Change this with:

```ruby
Ahoy.visitor_duration = 30.days
```


Question: How to: Token Generation

Answer:
Ahoy uses random UUIDs for visit and visitor tokens by default, but you can use your own generator like [ULID](https://github.com/rafaelsales/ulid).

```ruby
Ahoy.token_generator = -> { ULID.generate }
```


Question: How to: Throttling

Answer:
You can use [Rack::Attack](https://github.com/rack/rack-attack) to throttle requests to the API.

```ruby
class Rack::Attack
  throttle("ahoy/ip", limit: 20, period: 1.minute) do |req|
    if req.path.start_with?("/ahoy/")
      req.ip
    end
  end
end
```


Question: How to: Exceptions

Answer:
Exceptions are rescued so analytics do not break your app. Ahoy uses [Safely](https://github.com/ankane/safely) to try to report them to a service by default. To customize this, use:

```ruby
Safely.report_exception_method = ->(e) { Rollbar.error(e) }
```


Question: How to: Local Geocoding

Answer:
For privacy and performance, we recommend geocoding locally.

For city-level geocoding, download the [GeoLite2 City database](https://dev.maxmind.com/geoip/geolite2-free-geolocation-data).

Add this line to your application’s Gemfile:

```ruby
gem "maxminddb"
```

And create `config/initializers/geocoder.rb` with:

```ruby
Geocoder.configure(
  ip_lookup: :geoip2,
  geoip2: {
    file: "path/to/GeoLite2-City.mmdb"
  }
)
```

For country-level geocoding, install the `geoip-database` package. It’s preinstalled on Heroku. For Ubuntu, use:

```sh
sudo apt-get install geoip-database
```

Add this line to your application’s Gemfile:

```ruby
gem "geoip"
```

And create `config/initializers/geocoder.rb` with:

```ruby
Geocoder.configure(
  ip_lookup: :maxmind_local,
  maxmind_local: {
    file: "/usr/share/GeoIP/GeoIP.dat",
    package: :country
  }
)
```


Question: How to: Load Balancer Geocoding

Answer:
Some load balancers can add geocoding information to request headers.

- [nginx](https://nginx.org/en/docs/http/ngx_http_geoip_module.html)
- [Google Cloud](https://cloud.google.com/load-balancing/docs/custom-headers)
- [Cloudflare](https://support.cloudflare.com/hc/en-us/articles/200168236-Configuring-Cloudflare-IP-Geolocation)

Update `config/initializers/ahoy.rb` with:

```ruby
Ahoy.geocode = false

class Ahoy::Store < Ahoy::DatabaseStore
  def track_visit(data)
    data[:country] = request.headers["
"]
    data[:region] = request.headers["
"]
    data[:city] = request.headers["
"]
    super(data)
  end
end
```


Question: How to: IP Masking

Answer:
Ahoy can mask IPs with the same approach [Google Analytics uses for IP anonymization](https://support.google.com/analytics/answer/2763052). This means:

- For IPv4, the last octet is set to 0 (`8.8.4.4` becomes `8.8.4.0`)
- For IPv6, the last 80 bits are set to zeros (`2001:4860:4860:0:0:0:0:8844` becomes `2001:4860:4860::`)

```ruby
Ahoy.mask_ips = true
```

IPs are masked before geolocation is performed.

To mask previously collected IPs, use:

```ruby
Ahoy::Visit.find_each do |visit|
  visit.update_column :ip, Ahoy.mask_ip(visit.ip)
end
```


Question: How to: Anonymity Sets & Cookies

Answer:
Ahoy can switch from cookies to [anonymity sets](https://privacypatterns.org/patterns/Anonymity-set). Instead of cookies, visitors with the same IP mask and user agent are grouped together in an anonymity set.

```ruby
Ahoy.cookies = :none
```

Note: If Ahoy was installed before v5, [add an index](https://github.com/ankane/ahoy/tree/v5.0.0?tab=readme-ov-file#50) before making this change.

Previously set cookies are automatically deleted. If you use JavaScript tracking, also set:

```javascript
ahoy.configure({cookies: false});
```


Question: How to: Data Retention

Answer:
Data should only be retained for as long as it’s needed. Delete older data with:

```ruby
Ahoy::Visit.where("started_at < ?", 2.years.ago).find_in_batches do |visits|
  visit_ids = visits.map(&:id)
  Ahoy::Event.where(visit_id: visit_ids).delete_all
  Ahoy::Visit.where(id: visit_ids).delete_all
end
```

You can use [Rollup](https://github.com/ankane/rollup) to aggregate important data before you do.

```ruby
Ahoy::Visit.rollup("Visits", interval: "hour")
```

Delete data for a specific user with:

```ruby
user_id = 123
visit_ids = Ahoy::Visit.where(user_id: user_id).pluck(:id)
Ahoy::Event.where(visit_id: visit_ids).delete_all
Ahoy::Visit.where(id: visit_ids).delete_all
Ahoy::Event.where(user_id: user_id).delete_all
```


Question: How to: Development

Answer:
Ahoy is built with developers in mind. You can run the following code in your browser’s console.

Force a new visit

```javascript
ahoy.reset(); // then reload the page
```

Log messages

```javascript
ahoy.debug();
```

Turn off logging

```javascript
ahoy.debug(false);
```

Debug API requests in Ruby

```ruby
Ahoy.quiet = false
```


Question: How to: Track Additional Data

Answer:
```ruby
class Ahoy::Store < Ahoy::DatabaseStore
  def track_visit(data)
    data[:accept_language] = request.headers["Accept-Language"]
    super(data)
  end
end
```

Two useful methods you can use are `request` and `controller`.

You can pass additional visit data from JavaScript with:

```javascript
ahoy.configure({visitParams: {referral_code: 123}});
```

And use:

```ruby
class Ahoy::Store < Ahoy::DatabaseStore
  def track_visit(data)
    data[:referral_code] = request.parameters[:referral_code]
    super(data)
  end
end
```


Question: How to: Use Different Models

Answer:
```ruby
class Ahoy::Store < Ahoy::DatabaseStore
  def visit_model
    MyVisit
  end

  def event_model
    MyEvent
  end
end
```


Question: How to: Explore the Data

Answer:
[Blazer](https://github.com/ankane/blazer) is a great tool for exploring your data.

With Active Record, you can do:

```ruby
Ahoy::Visit.group(:search_keyword).count
Ahoy::Visit.group(:country).count
Ahoy::Visit.group(:referring_domain).count
```

[Chartkick](https://www.chartkick.com/) and [Groupdate](https://github.com/ankane/groupdate) make it easy to visualize the data.

```erb
<%= line_chart Ahoy::Visit.group_by_day(:started_at).count %>
```


Question: How to: Querying Events

Answer:
Ahoy provides a few methods on the event model to make querying easier.

To query on both name and properties, you can use:

```ruby
Ahoy::Event.where_event("Viewed product", product_id: 123).count
```

Or just query properties with:

```ruby
Ahoy::Event.where_props(product_id: 123, category: "Books").count
```

Group by properties with:

```ruby
Ahoy::Event.group_prop(:product_id, :category).count
```

Note: MySQL and MariaDB always return string keys (including `"null"` for `nil`) for `group_prop`.


Question: How to: Forecasting

Answer:
To forecast future visits and events, check out [Prophet](https://github.com/ankane/prophet).

```ruby
daily_visits = Ahoy::Visit.group_by_day(:started_at).count # uses Groupdate
Prophet.forecast(daily_visits)
```


Question: How to: Anomaly Detection

Answer:
To detect anomalies in visits and events, check out [AnomalyDetection.rb](https://github.com/ankane/AnomalyDetection.rb).

```ruby
daily_visits = Ahoy::Visit.group_by_day(:started_at).count # uses Groupdate
AnomalyDetection.detect(daily_visits, period: 7)
```


Question: How to: Breakout Detection

Answer:
To detect breakouts in visits and events, check out [Breakout](https://github.com/ankane/breakout).

```ruby
daily_visits = Ahoy::Visit.group_by_day(:started_at).count # uses Groupdate
Breakout.detect(daily_visits)
```


Question: How to: Recommendations

Answer: To make recommendations based on events, check out [Disco](https://github.com/ankane/disco#ahoy).


Question: How to: Contributing

Answer:
Everyone is encouraged to help improve this project. Here are a few ways you can help:

- [Report bugs](https://github.com/ankane/ahoy/issues)
- Fix bugs and [submit pull requests](https://github.com/ankane/ahoy/pulls)
- Write, clarify, or fix documentation
- Suggest or add new features

To get started with development:

```sh
git clone https://github.com/ankane/ahoy.git
cd ahoy
bundle install
bundle exec rake test
```

To test different adapters, use:

```sh
ADAPTER=postgresql bundle exec rake test
ADAPTER=mysql2 bundle exec rake test
ADAPTER=mongoid bundle exec rake test
```


Question: How to: Awesome Apache Airflow

Answer:
![contrib badge](https://img.shields.io/github/contributors/jghoman/awesome-apache-airflow.svg) ![GitHub commit activity](https://img.shields.io/github/commit-activity/m/jghoman/awesome-apache-airflow?style=plastic)

This is a curated list of resources about [Apache Airflow](https://airflow.apache.org/).  Please feel free to contribute any items that should be included. Items are generally added at the top of each section so that more fresh items are featured more prominently.


Question: How to: Vital links

Answer:
- [Source code](https://github.com/apache/airflow/) (latest stable release [1.10.12](https://github.com/apache/airflow/tree/1.10.12))
- [Documentation](https://airflow.apache.org/) (also the official website)
- [Confluence page](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Home)
- [![Twitter Follow](https://img.shields.io/twitter/follow/apacheairflow?style=social)](https://twitter.com/ApacheAirflow)
- [Slack workspace](https://apache-airflow-slack.herokuapp.com/)


Question: How to: Airflow deployment solutions

Answer:
- [Installing Airflow on IBM Cloud](https://github.com/KissConsult/Apache-Airflow) - Quick and easy deployment on IBM Cloud with IBM [Bitnami Charts](https://github.com/bitnami/charts)
- [Three ways to run Airflow on Kubernetes](https://fullstaq.com/blog/three-ways-to-run-airflow-on-kubernetes/) - [Tim van de Keer](https://www.linkedin.com/in/tim-van-de-keer-bb5a1966) walks through several methods for deploying Airflow on Kubernetes.
- [Apache Airflow Multi-Tier Free Deployment on Azure](https://azure.microsoft.com/en-us/blog/bitnami-apache-airflow-multi-tier-now-available-in-azure-marketplace/) - A free Azure Resource Manager (ARM) template by Bitnami providing a one-click solution for Airflow deployment on Azure for production use-cases.
- [KubernetesExecutor Helm Chart](https://github.com/tekn0ir/airflow-chart) - A lean Helm Chart using the KubernetesExecutor for a more k8s native experience and complementary [KubernetesExecutor Docker Image](https://github.com/tekn0ir/airflow-docker).
- [Stable Celery Helm Chart](https://github.com/helm/charts/tree/master/stable/airflow) - Curated Helm Chart in the official stable chart repository.
- [Puckel's Docker Image](https://github.com/puckel/docker-airflow) - [@Puckel_](https://twitter.com/Puckel_)'s well-crafted Docker image has become the base for many Airflow installations.  It is regularly updated and closely tracks the official Apache releases.
- [Kubernetes Custom Operator for Deploying Airflow](https://github.com/GoogleCloudPlatform/airflow-operator) - Kubernetes Custom controller (also called operator pattern) for deploying Airflow on Kubernetes.
- [airflow-pipeline](https://github.com/datagovsg/airflow-pipeline) - Airflow Docker container that comes preconfigured for Spark and Hadoop.  It can be docker pulled at ``datagovsg/airflow-pipeline``.
- [aws-airflow-stack](https://github.com/villasv/aws-airflow-stack) - An AWS based Airflow cluster deployment with CeleryExecutor. Deploys after a few clicks with CloudFormation.
- [kube-airflow](https://github.com/mumoshu/kube-airflow) - This repository contains both an Airflow Docker image (that appears to have been based on Puckel's work) and Kubernetes service definition.  [mumoshu](https://github.com/mumoshu)'s repository has not been recently updated, but there are numerous forks that may be based on more recent releases.
- [airflow-on-kubernetes](https://github.com/rolanddb/airflow-on-kubernetes) - A guide on all relevant resources, scripts and projects that relate to running Airflow on Kubernetes.
- [airflow-k8s-executor-on-GKE](https://github.com/EamonKeane/airflow-GKE-k8sExecutor-helm) - A detailed tutorial to get a scalable, low maintenance airflow kubernetes executor environment deployed on [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/) with [helm](https://helm.sh/).
- [airflow-cookbook](https://github.com/bahchis/airflow-cookbook) - Chef cookbook for deploying Airflow.
- [Running Airflow on top of Apache Mesos](http://agrajmangal.in/blog/big-data/running-airflow-on-top-of-apache-mesos/) - Blog describing how to configure [Mesos](http://mesos.apache.org/) to run all of the Airflow componenents.
- [Integrating Apache Airflow with Apache Ambari](https://medium.com/@mykolamykhalov/integrating-apache-airflow-with-apache-ambari-ccab2c90173) - [Mykola Mykhalov](https://www.linkedin.com/in/mykola-mykhalov-9079a8107/) walks through using [Apache Ambari](https://ambari.apache.org/) to configure and deploy an Airflow instance.
- [Astronomer Platform](https://github.com/astronomerio/astronomer) - Apache Airflow as a Service on Kubernetes. For more information visit https://www.astronomer.io.
- [Bitnami Airflow Docker image](https://github.com/bitnami/bitnami-docker-airflow) - A secure and up-to-date docker image for Airflow maintained by Bitnami.
- [Bitnami Airflow Scheduler Docker image](https://github.com/bitnami/bitnami-docker-airflow-scheduler) - A secure and up-to-date docker image for Airflow Scheduler maintained by Bitnami.
- [Bitnami Airflow Worker Docker image](https://github.com/bitnami/bitnami-docker-airflow-worker) - A secure and up-to-date docker image for Airflow Worker maintained by Bitnami. A CeleryExecutor docker-compose deployment is available [here](https://github.com/bitnami/bitnami-docker-airflow-worker/blob/master/docker-compose.yml).
- [Distribute & deploy Apache Airflow via Python PEX files](https://github.com/msumit/airflow-pex) - Example repo with steps to bundle, distribute, & deploy Apache Airflow as PEX files.
- [Introducing KEDA for Airflow](https://www.astronomer.io/blog/the-keda-autoscaler/) - How to use KEDA scaler system to enable autoscaling of celery workers based on data stored in the Airflow metadata database.
- [Airflow-Component](https://github.com/noelmcloughlin/airflow-component#lightweight-federated-apache-airflow-installer) - Lightweight installer of federated Airflow-Airflow (RabbitMQ) reference architectrure on Compute node(s).


Question: How to: Introductions and tutorials

Answer:
- [Apache Airflow Monitoring Metrics](https://youtu.be/xyeR_uFhnD4) - A two-part series by [maxcotec](https://maxcotec.com) on how you can utilize existing Airflow statsd metrics to monitor your airflow deployment on Grafana dashboard via Prometheus. Also learn how to create custom metrics.
- [Introduction to Airflow](https://www.youtube.com/playlist?list=PLzKRcZrsJN_xcKKyKn18K7sWu5TTtdywh) - A web tutorial series by [maxcotec](https://maxcotec.com) for beginners and intermediate users of Apache Airflow.
- [ETL with Apache Airflow for Data Analysis on Transaction Data](https://github.com/KimaruThagna/ml-pipelines-airflow). [Kimaru Thagana](https://www.linkedin.com/in/kimaru-thagana-4920b5181/) covers a practical case of doing an ETL process using Apache Airflow using a dummy ecommerce store's transactional, user and product data. The data is served via a flask API.
- [Start Building Better Data Pipelines With apache Airflow](https://blog.delaplex.com/start-building-better-data-pipelines-with-apache-airflow) 2020-Oct - Naman Gupta covers the basics of Airflow and its concepts.
- [Airflow Repository Template](https://github.com/soggycactus/airflow-repo-template) - A boilerplate repository for developing locally with Airflow, with linting & tests for valid DAGs and plugins. Just clone and run `make start-airflow` to get started! Add some CI jobs to deploy your code and you're done.
- [How Apache Airflow Distributes Jobs on Celery workers](https://blog.sicara.com/using-airflow-with-celery-workers-54cb5212d405) - A short description of the steps taken by a task instance, from scheduling to success, in a distributed architecture.
- [Remote spark-submit to YARN running on EMR](https://medium.com/@tamizhgeek/remote-spark-submit-toyarn-running-on-emr-9804b89d82d2) - [Azhaguselvan](https://github.com/tamizhgeek) walks through submitting Spark jobs to existing EMR clusters with Airflow.
- [Running Airflow on top of Apache Mesos](http://agrajmangal.in/blog/big-data/running-airflow-on-top-of-apache-mesos/) and its follow-up, [Mesos, Airflow & Docker](http://agrajmangal.in/blog/big-data/mesos-airflow-docker/) by [Agraj Mangal](https://twitter.com/agrajm) is a quick overview of running Airflow atop Apache Mesos.
- [Dustin Stansbury](https://twitter.com/corrcoef) of [Quizlet](https://quizlet.com/) has written a four-part series that covers what workflow managers do in general, how Quizlet picked Airflow, a tour of Airflow's key concepts, and how Quizlet is now using Airflow in practice:
  - [Beyond CRON: an introduction to Workflow Management Systems](https://medium.com/@dustinstansbury/beyond-cron-an-introduction-to-workflow-management-systems-19987afcdb5e)
  - [Why Quizlet chose Apache Airflow for executing data workflows](https://towardsdatascience.com/why-quizlet-chose-apache-airflow-for-executing-data-workflows-3f97d40e9571)
  - [Understanding Apache Airflow’s key concepts](https://medium.com/@dustinstansbury/understanding-apache-airflows-key-concepts-a96efed52b1a)
  - [How Quizlet uses Apache Airflow in practice](https://medium.com/@dustinstansbury/how-quizlet-uses-apache-airflow-in-practice-a903cbb5626d)
- [Integrating Apache Airflow with Databricks](https://databricks.com/blog/2017/07/19/integrating-apache-airflow-with-databricks.html) - While this tutorial is focused specifically on Databricks' Spark solutions, it does have a reasonable overview of Airflow basics and demonstrates how a third party solution can quickly integrate into Airflow.
- [Apache Airflow 2.0 Tutorial](https://turbaszek.medium.com/apache-airflow-2-0-tutorial-41329bbf7211) - This article discusses the basic concepts that stand behind Airflow and discusses the problems it solves.
- [Testing and debugging Apache Airflow](https://blog.godatadriven.com/testing-and-debugging-apache-airflow) - Article explaining how to apply unit testing, mocking and debugging to Airflow code.
- [Get started developing workflows with Apache Airflow](http://michal.karzynski.pl/blog/2017/03/19/developing-workflows-with-apache-airflow/) - This brief introductory tutorial covers how to create data pipeline and processing workflow using DAG, operators, Sensor, using Xcoms to communicate between operators.
- [Get started with Airflow + Google Cloud Platform + Docker](https://medium.com/@junjiejiang94/get-started-with-airflow-google-cloud-platform-docker-a21c46e0f797) - Step-by-step introduction by [Jayce Jiang](https://medium.com/@junjiejiang94).
- [How to develop data pipeline in Airflow through TDD (test-driven development)](https://blog.magrathealabs.com/how-to-develop-data-pipeline-in-airflow-through-tdd-test-driven-development-c3333439f358) - Learn how to build a sales data pipeline using TDD step-by-step and in the end how to configure a simple CI workflow using Github Actions.


Question: How to: Airflow Summit 2020 videos

Answer:
*The first [Airflow Summit 2020](https://airflowsummit.org/) was held in July 2020. It was a truly global,
fully online event that was co-hosted by 9 Airflow Meetups from all over the world ([Melbourne](https://www.meetup.com/Melbourne-Apache-Airflow-Meetup/),
[Tokyo](https://www.meetup.com/Tokyo-Apache-Airflow-incubating-Meetup),
[Bangalore](https://www.meetup.com/Bangalore-Apache-Airflow-Meetup/),
[Warsaw](https://www.meetup.com/Warsaw-Airflow-Meetup/),
[Amsterdam](https://www.meetup.com/Amsterdam-Airflow-meetup/),
[London](https://www.meetup.com/London-Apache-Airflow-Meetup/),
[NYC](https://www.meetup.com/NYC-Apache-Airflow-Meetup/),
[BayArea](https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/)).*

*It featured 40+ talks and three workshops. You can check out the talk recordings as a YouTube
[Airflow Summit 2020 Playlist](https://www.youtube.com/playlist?list=PLGudixcDaxY3RGLSlWoN_cEEXhIT1OPmj)
or see the individual talks here:*

- [Keynote: Airflow then and now](https://youtu.be/GB2f7ZhRCho) ![Activity badge](https://img.shields.io/youtube/views/GB2f7ZhRCho)
- [Scheduler as a service - Apache Airflow at EA Digital Platform](https://youtu.be/u00wmcHe8ow) ![Activity badge](https://img.shields.io/youtube/views/u00wmcHe8ow)
- [Keynote: How large companies use Airflow for ML and ETL pipelines](https://youtu.be/428AiCBMZoQ) ![Activity badge](https://img.shields.io/youtube/views/428AiCBMZoQ)
- [Data DAGs with lineage for fun and for profit](https://youtu.be/l_vVxOdvujg) ![Activity badge](https://img.shields.io/youtube/views/l_vVxOdvujg)
- [Airflow on Kubernetes: Containerizing your workflows](https://youtu.be/3VDeKmxHWYA) ![Activity badge](https://img.shields.io/youtube/views/3VDeKmxHWYA)
- [Data flow with Airflow @ PayPal](https://youtu.be/kAtaj_s4f-w) ![Activity badge](https://img.shields.io/youtube/views/kAtaj_s4f-w)
- [Democratised data workflows at scale](https://youtu.be/Cd4-YtHYT9M) ![Activity badge](https://img.shields.io/youtube/views/Cd4-YtHYT9M)
- [Migrating Airflow-based Spark jobs to Kubernetes - the native way](https://youtu.be/i79OsoLUx0k) ![Activity badge](https://img.shields.io/youtube/views/i79OsoLUx0k)
- [Keynote: Future of Airflow](https://youtu.be/YLsGVFB8Pws) ![Activity badge](https://img.shields.io/youtube/views/YLsGVFB8Pws)
- [Run Airflow DAGs in a secure way](https://youtu.be/QhnItssm4yU) ![Activity badge](https://img.shields.io/youtube/views/QhnItssm4yU)
- [Keynote: Making Airflow a sustainable project through D&I](https://youtu.be/wxn9ta13Gbo) ![Activity badge](https://img.shields.io/youtube/views/wxn9ta13Gbo)
- [Airflow CI/CD: Github to Cloud Composer (safely)](https://youtu.be/ZgTf523XM0g) ![Activity badge](https://img.shields.io/youtube/views/ZgTf523XM0g)
- [Advanced Apache Superset for Data Engineers](https://youtu.be/Mhai7sVU244) ![Activity badge](https://img.shields.io/youtube/views/Mhai7sVU244)
- [Demo: Reducing the lines, a visual DAG editor](https://youtu.be/I4nFCqEnOJc) ![Activity badge](https://img.shields.io/youtube/views/I4nFCqEnOJc)
- [AIP-31: Airflow functional DAG definition](https://youtu.be/II4Ip81T3qc) ![Activity badge](https://img.shields.io/youtube/views/II4Ip81T3qc)
- [Autonomous driving with Airflow](https://youtu.be/wEq1FGe6oBY) ![Activity badge](https://img.shields.io/youtube/views/wEq1FGe6oBY)
- [From cron to Airflow on Kubernetes: A startup story](https://youtu.be/giQReCd7jp8) ![Activity badge](https://img.shields.io/youtube/views/giQReCd7jp8)
- [Achieving Airflow Observability](https://youtu.be/Hc4pYAUL6Qs) ![Activity badge](https://img.shields.io/youtube/views/Hc4pYAUL6Qs)
- [Machine Learning with Apache Airflow](https://youtu.be/N_3RQeqySE0) ![Activity badge](https://img.shields.io/youtube/views/N_3RQeqySE0)
- [Airflow: A beast character in the gaming world](https://youtu.be/BH0ut33zp9A) ![Activity badge](https://img.shields.io/youtube/views/BH0ut33zp9A)
- [Effective Cross-DAG dependency](https://youtu.be/p66GcO0LbFQ) ![Activity badge](https://img.shields.io/youtube/views/p66GcO0LbFQ)
- [What open source taught us about business](https://youtu.be/KIEMEYM2PEs) ![Activity badge](https://img.shields.io/youtube/views/KIEMEYM2PEs)
- [Data engineering hierarchy of needs](https://youtu.be/SvnTyDiZOzQ) ![Activity badge](https://img.shields.io/youtube/views/SvnTyDiZOzQ)
- [Building reuseable and trustworthy ELT pipelines (A templated approach)](https://youtu.be/R4bp3_VyJ70) ![Activity badge](https://img.shields.io/youtube/views/R4bp3_VyJ70)
- [Testing Airflow workflows - ensuring your DAGs work before going into production](https://youtu.be/ANJnYbLwLjE) ![Activity badge](https://img.shields.io/youtube/views/ANJnYbLwLjE)
- [Adding an executor to Airflow: A contributor overflow exception](https://youtu.be/RKEmAshcreE) ![Activity badge](https://img.shields.io/youtube/views/RKEmAshcreE)
- [Migration to Airflow backport providers](https://youtu.be/1SSlxAcOEso) ![Activity badge](https://img.shields.io/youtube/views/1SSlxAcOEso)
- [From Zero to Airflow: b


Question: How to: Best practices, lessons learned and cool use cases

Answer:
- [How to Best Use DuckDB with Apache Airflow](https://medium.com/apache-airflow/how-to-best-use-duckdb-with-apache-airflow-63a079160d5d) - Tips on integrating [DuckDB](https://duckdb.org/) into Airflow jobs.
- [Airflow Dag Python Package Management](https://www.youtube.com/watch?v=9pykChPp-X4&t=121s) - Managing python package dependencies across 100+ dags can become painful. It's hard to keep track of which packages are used by which dag, and hard to clean up during DAG removal/upgrade. Learn how KubernetesPodOperator and DockerOperator can fix this.
- [Airflow Dag Management & Versioning](https://youtu.be/a-4yRne3ba4) - Efficently manage DAGs release process by using Git Submodules
- [Testing in Airflow Part 2](https://medium.com/@chandukavar/testing-in-airflow-part-2-integration-tests-and-end-to-end-pipeline-tests-af0555cd1a82) - [Chandu Kavar](https://twitter.com/chandukavar) and [Sarang Shinde](https://www.linkedin.com/in/sarang-shinde-219a4873/) have explained Integration Tests and End-to-End Pipeline Tests.
- [Upgrading & Scaling Airflow at Robinhood](https://robinhood.engineering/upgrading-scaling-airflow-at-robinhood-5b625dfaa2ee) - [Abishek Ray](https://www.linkedin.com/in/abhishek-ray-29210145/) describes how Robinhood tackled upgrading its production Airflow while minimizing downtime.
- [We're all using Airflow wrong and how to fix it](https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-and-how-to-fix-it-a56f14cb0753) - [Jessica Laughlin](https://www.jldlaughlin.com/) of [Bluecore](https://www.bluecore.com/) shares three engineering problems associated with the Airflow design and how to solve them by using the [KubernetesPodOperator](https://github.com/apache/airflow/blob/v1-10-stable/airflow/contrib/operators/kubernetes_pod_operator.py) in two design patterns.
- [Getting started with Data Lineage](https://medium.com/dailymotion/getting-started-with-data-lineage-6307b2b429b3) - [Germain Tanguy](https://www.linkedin.com/in/germain-tanguy/) of [Dailymotion](https://www.dailymotion.com/) shares a data lineage prototype integrated to Apache Airflow.
- [Collaboration between data engineers, data analysts and data scientists](https://medium.com/dailymotion/collaboration-between-data-engineers-data-analysts-and-data-scientists-97c00ab1211f) - [Germain Tanguy](https://www.linkedin.com/in/germain-tanguy/) of [Dailymotion](https://www.dailymotion.com/) shares how to efficiently release in production by collaboration with Apache Airflow.
- [Using Apache Airflow’s Docker Operator with Amazon’s Container Repository](https://www.lucidchart.com/techblog/2019/03/22/using-apache-airflows-docker-operator-with-amazons-container-repository/) - [Brian Campbell](https://www.linkedin.com/in/bvcampbell3) of [Lucid](https://www.lucidchart.com/) has tips for integrating AWS's [ECR](https://aws.amazon.com/ecr/) service with Airflow's DockerOperator.
- [Airflow: Lesser Known Tips, Tricks, and Best Practises](https://medium.com/datareply/airflow-lesser-known-tips-tricks-and-best-practises-cf4d4a90f8f) - [Kaxil Naik](https://www.linkedin.com/in/kaxil/) has explained the lesser-known yet very useful tips and best practises on using Airflow.
- [boundary-layer:Declarative Airflow Workflows](https://codeascraft.com/2018/11/14/boundary-layer%E2%80%89-declarative-airflow-workflows/) - [Kevin McHale](https://www.linkedin.com/in/mchalek) has explained open source project boundary-layer which generates airflow dag with declarative workflows.
- [Testing in Airflow Part 1](https://medium.com/@chandukavar/testing-in-airflow-part-1-dag-validation-tests-dag-definition-tests-and-unit-tests-2aa94970570c) - [Chandu Kavar](https://twitter.com/chandukavar) has explained different categories of tests in Airflow. It includes DAG Validation Tests, DAG Definition Tests, and unit tests.
- [Improving Airflow UI Security](https://wecode.wepay.com/posts/improving-airflow-ui-security) - WePay's [Joy Gao](https://twitter.com/joygao) breaks down the need for Role Based Access Controls (RBAC) and how she introduced it to Airflow.
- [How to Create a Workflow in Apache Airflow to Track Disease Outbreaks in India](https://blog.socialcops.com/engineering/apache-airflow-disease-outbreaks-india/) - [Vinayak Mehta](https://twitter.com/vortex_ape) details how [SocialCops](https://socialcops.com/) uses Airflow to scrape India's Ministry of Health and Family Affairs to generate derived data on possible disease outbreaks.
- [Airflow, Meta Data Engineering, and a Data Platform for the World’s Largest Democracy](https://blog.socialcops.com/technology/engineering/airflow-meta-data-engineering-disha/) - [Vinayak Mehta](https://twitter.com/vortex_ape) talks about identifying data engineering patterns (meta data engineering) to automate DAG generation and how that helped [SocialCops](https://socialcops.com/) to power DISHA, a national data platform where Indian MPs and MLAs monitor the progress of 42 national level schemes.
- [Lessons learnt while Airflo


Question: How to: Books, blogs, podcasts, and such

Answer:
- [Data Pipelines with Apache Airflow](https://www.manning.com/books/data-pipelines-with-apache-airflow) - A Manning book (Early Access September 2019) on Airflow.
- [The Airflow Podcast](https://soundcloud.com/the-airflow-podcast) - A semiregular podcast discussing all things Airflow.
- [Maxime Beauchemin](https://medium.com/@maximebeauchemin) - Maxime's blog on medium that gives insight into the philosophy behind Apache Airflow.
- [Robert Chang](https://medium.com/@rchang) - Blog posts about data engineering with Apache Airflow, explains why and has examples in code.
- [Handling Airflow logs with Kubernetes Executor](https://szeevs.medium.com/handling-airflow-logs-with-kubernetes-executor-25c11ea831e4) - A blogpost that outlines how you can set up remote S3 logging when using KubernetesExecutor, without creating complex infrastructure.
- [Airflow 2.0: DAG Authoring Redesigned](https://turbaszek.medium.com/airflow-2-0-dag-authoring-redesigned-651edc397178) - Blog post about new ways of writing DAGs in Airflow 2.0.
- [Airflow 2.0 Providers](https://higrys.medium.com/airflow-2-0-providers-1bd21ba3bd93) - Blog post about providers packages in Airflow 2.0.


Question: How to: Slide deck presentations and online videos

Answer:
- 2020-Feb: [Apache Airflow @ Umuzi.org](https://www.youtube.com/watch?IAmWKZDmvek) ![Activity badge](https://img.shields.io/youtube/views/IAmWKZDmvek) - [Sheena O'Connell](https://twitter.com/sheena_oconnell) discusses how South Africa-based tech bootcamp [Umuzi](https://www.umuzi.org/) uses Airflow.
- [Apache Airflow YouTube tutorials](https://www.youtube.com/playlist?list=PL79i7SgJCJ9hu5GqcA091h6zuewmsvSyy) - [Marc Lamberti](https://twitter.com/marclambertiml) has created a series of YouTube tutorials covering many aspects of Airflow concepts, configuration and deployment.
- [Advanced Data Engineering Patterns with Apache Airflow](https://www.youtube.com/watch?23_1WlxGGM4) ![Activity badge](https://img.shields.io/youtube/views/23_1WlxGGM4) - Video of [Maxime Beauchemin](https://medium.com/@maximebeauchemin)'s talk that briefly introduces Airflow and then goes into more advanced use cases, including self-servive SQL queries, building A/B testing metrics frameworks and machine learning feature extraction all via Airflow.  The slides are available separately [here](https://prezi.com/p/adxlaplcwzho/advanced-data-engineering-patterns-with-apache-airflow/).
- [Modern Data Pipelines with Apache Airflow](https://blog.tedmiston.com/momentum-2018-airflow-talk/) - A talk given by [Taylor Edmiston](https://twitter.com/kicksopenminds) and [Andy Cooper](https://twitter.com/andscoop) from Astronomer.io at Momentum Dev Con 2018 on getting started with Airflow, custom components, example DAGs, and the Astronomer Airflow CLI.
- [Building Better Data Pipelines using Apache Airflow](https://www.slideshare.net/r39132/building-better-data-pipelines-using-apache-airflow-94060954) - Slides from [Sid Anand](https://twitter.com/r39132)'s talk at QCon 18 with a thorough overview of Airflow and its architecture.
- [Airflow and Spark Streaming at Astronomer](https://paper.dropbox.com/doc/Airflow-Spark-talk-v2.0-5own4Nlz8MhdwKQ8QhIqj?_tk=share_copylink) - How Astronomer uses dynamic DAGs to run Spark Streaming jobs with Airflow.
- [Apache Airflow in the Cloud: Programmatically orchestrating workloads with Python](https://www.slideshare.net/kaxil/apache-airflow-in-the-cloud-programmatically-orchestrating-workloads-with-python-pydata-london-2018-95391267) - Slides from [Kaxil Naik](https://twitter.com/kaxil)'s & [Satyasheel](https://twitter.com/ss6012) talk at PyData London 18 introducing the basics of Airflow and how to orchestrate workloads on Google Cloud Platform (GCP).
- [Developing elegant workflows in Python code with Apache Airflow](https://eventil.com/presentations/j2sK9R) - [Michał Karzyński](https://twitter.com/postrational) at [Europython](https://ep2018.europython.eu/) gives a brief introduction to Airflow concepts including the role of workflow managers, DAGs and operators.  Link includes both video and slides.
- [Data Pipeline Management](https://www.youtube.com/watch?mjn1LAZ_Y38) ![Activity badge](https://img.shields.io/youtube/views/mjn1LAZ_Y38) - [Ben Goldberg](https://www.linkedin.com/in/benjamin-goldberg-50247169/) walks the Chicago Kubernetes Meetup through how [SpotHero](https://spothero.com/) uses Airflow. Additionally, Ben has a very [complete slidedeck](https://docs.google.com/presentation/d/1hc12cFs5TmEajLwYNASLwz_C17Q5tyd__6oXzI16A9A/edit#slide=id.g320d39a12c_0_1017) of how Airflow plays within Kubernetes.
- [How I learned to time travel, or, data pipelining and scheduling with Airflow](https://www.slideshare.net/LauraLorenz4/how-i-learned-to-time-travel-or-data-pipelining-and-scheduling-with-airflow) - Comprehensive deck by [Laura Lorenz](https://twitter.com/lalorenz6) for why Airflow is necessary and how [Industry Dive](https://www.industrydive.com/) uses it.
- [Introduction to Apache Airflow - Data Day Seattle 2016](https://www.slideshare.net/r39132/introduction-to-apache-airflow-data-day-seattle-2016) - [Sid Anand](https://twitter.com/r39132) gives a thorough introduction to Airflow and how it was used at [Agari](https://www.agari.com/).
- [Operating Data Pipeline With Airflow - Airflow Meetup April-2018](https://speakerdeck.com/vananth22/operating-data-pipeline-with-airflow-at-slack) - [Ananth Packkildurai](https://twitter.com/ananthdurai) talks about scaling airflow Local Executor and best practices to operate data pipeline at [Slack](https://slack.com/).
- [Apache Airflow at WePay](https://wepayinc.app.box.com/s/hf1chwmthuet29ux2a83f5quc8o5q18k) - [Chris Riccomini](https://twitter.com/criccomini) discusses why WePay chose Airflow and provides a detailed breakdown of their deployment and the infrastructure behind it.
- [Elegant data pipelining with Apache Airflow](https://www.youtube.com/watch?v=neuh_2_zrt8) - Talks from [Bolke de Bruin](https://twitter.com/bolke2028) and [Fokko Driesprong](https://twitter.com/fokkodriesprong) at PyData Amsterdam 2018 about methodologies that provide clarity in ETL using Airflow.
- [Airflow @ Lyft](https://www.slideshare.net/taofung/airflow-at-lyft) - Talks from [Tao Fe


Question: How to: Libraries, Hooks, Utilities

Answer:
- [Domino](https://github.com/Tauffer-Consulting/domino) - Domino is an open source Graphical User Interface platform for creating data and Machine Learning workflows (DAGs) with no-code, visually intuitive drag-and-drop actions. It is also a standard for publishing and sharing your Python code so it can be automatically used by anyone, directly in the GUI.
- [Airflow-Helper](https://github.com/xnuinside/airflow-helper) - setting up Airflow Variables, Connections, and Pools from a YAML configuration file.
- [AirFly](https://github.com/ryanchao2012/airfly) - Auto generate Airflow's dag.py on the fly.
- [DEAfrica Airflow](https://github.com/digitalearthafrica/deafrica-airflow) - Airflow libraries used by [Digital Earth Africa](https://digitalearthafrica.org/), an humanitarian effort to utilize satellite imagery of Africa.
- [Airflow plugins](https://github.com/airflow-plugins/) - Central collection of repositories of various plugins for Airflow, including mailchimp, trello, sftp, GitHub, etc.
- [fileflow](https://github.com/industrydive/fileflow) - Collection of modules to support large data transfers between Airflow operators through either local file system or S3.  This addresses a gap where data is too large for XCOMs but too small or inconvenient for loading directly in the operator.  Built by [Industry Dive](https://www.industrydive.com/).
- [fairflow](https://github.com/michaelosthege/fairflow) - Library to abstract away Airflow's Operators with functional pieces that transform the data from one operator to another.
- [airflow-maintenance-dags](https://github.com/teamclairvoyant/airflow-maintenance-dags) - [Clairvoyant](http://clairvoyantsoft.com/) has a repo of Airflow DAGs that operator on Airflow itself, clearing out various bits of the backing metadata store.
- [test_dags](https://gist.github.com/criccomini/2862667822af7fae8b55682faef029a7) - a more complete solution for DAG integrity tests ([first Circle of Data’s Inferno are the first](https://medium.com/@ingwbaa/datas-inferno-7-circles-of-data-testing-hell-with-airflow-cef4adff58d8).
- [dag-factory](https://github.com/ajbosco/dag-factory) - A library for dynamically generating Apache Airflow DAGs from YAML configuration files.
- [whirl](https://github.com/godatadriven/whirl) - Fast iterative local development and testing of Apache Airflow workflows.
- [airflow-code-editor](https://github.com/andreax79/airflow-code-editor) - A plugin for Apache Airflow that allows you to edit DAGs in browser.
- [Pylint-Airflow](https://github.com/BasPH/pylint-airflow) - A Pylint plugin for static code analysis on Airflow code.
- [afctl](https://github.com/qubole/afctl) - A CLI tool that includes everything required to create, manage and deploy airflow projects faster and smoother.
- [Dag Dependencies viewer](https://github.com/ms32035/airflow-dag-dependencies) - A plugin which creates a view to visualize dependencies between the Airflow DAGs
- [Airflow ECR Plugin](https://github.com/asandeep/airflow-ecr-plugin) - Plugin to refresh AWS ECR login token at regular intervals. This is helpful where DockerOperator needs to pull images hosted on ECR.
- [AirflowK8sDebugger](https://github.com/Javier162380/AirflowKuberentesDebugger) - A library for generate k8s pod yaml templates from an Airflow dag using the KubernetesPodOperator.
- [Oozie to Airflow](https://github.com/GoogleCloudPlatform/oozie-to-airflow) - A tool to easily convert between [Apache Oozie](http://oozie.apache.org/) workflows and Apache Airflow workflows.
- [Airflow Ditto](https://github.com/angadsingh/airflow-ditto) - An extensible framework to do transformations to an Airflow DAG and convert it into another DAG which is flow-isomorphic with the original DAG, to be able to run it on different environments (e.g. on different clouds, or even different container frameworks - Apache Spark on YARN vs Kubernetes). Comes with out-of-the-box support for EMR-to-HDInsight-DAG transforms.
- [gusty](https://github.com/chriscardillo/gusty) - Create a DAG using any number of YAML, Python, Jupyter Notebook, or R Markdown files that represent individual tasks in the DAG. gusty also configures dependencies, DAGs, and TaskGroups, features support for your local operators, and more. A fully containerized demo is available [here](https://github.com/chriscardillo/gusty-demo).
- [Meltano](https://www.meltano.com) - Open source, self-hosted, CLI-first, debuggable, and extensible ELT tool that embraces [Singer](https://www.singer.io) for extraction and loading, leverages [dbt](https://www.getdbt.com) for transformation, and [integrates with Airflow for orchestration](https://meltano.com/#orchestration).
- [DAG checks](https://github.com/politools/dag-checks) - The dag-checks consist of checks that can help you in maintaining your Apache Airflow instance.
- [Airflow DVC plugin](https://github.com/covid-genomics/airflow-dvc) - Plugin for open-source version-control system for data science and Machine Learning pipelines - [DVC](htt


Question: How to: Commercial Airflow-as-a-service providers

Answer:
- [Google Cloud Composer](https://cloud.google.com/composer/) - Google Cloud Composer is a managed service built atop Google Cloud and Airflow.
- [Qubole](http://docs.qubole.com/en/latest/user-guide/airflow/) - Qubole is mainly known as a service-and-support company for Apache Hive, but also provides Airflow as a component of its platform.
- [Astronomer.io](https://www.astronomer.io/) - Astronomer provides complete ETL lifecycle solutions and appears to be entirely focused on providing Airflow-based products.
- [AWS MWAA](https://aws.amazon.com/managed-workflows-for-apache-airflow/) - Amazon Managed Workflows for Apache Airflow (MWAA) is a managed orchestration service for Apache Airflow that makes it easier to set up and operate end-to-end data pipelines in the cloud at scale.


Question: How to: Cloud Composer resources

Answer:
*This section contains articles that apply to [Cloud Composer](https://cloud.google.com/composer) — a service built by Google Cloud based on Apache Airflow. Tricks and solutions are described here that are intended for Cloud Composer, but may be applicable to vanilla Airflow.*

- [Enabling Autoscaling in Google Cloud Composer](https://medium.com/traveloka-engineering/enabling-autoscaling-in-google-cloud-composer-ac84d3ddd60) - Supercharge your Cloud Composer deployment while saving up some cost during idle periods.
- [Scale your Composer environment together with your business](https://cloud.google.com/blog/products/data-analytics/scale-your-composer-environment-together-your-business) - The Celery Executor architecture and ways to ensure high scheduler performance.
- [pianka.sh](https://github.com/politools/airflow-pianka-sh) - Missing command in the gcloud tool. This tool facilitates some administrative tasks.
- [The Smarter Way of Scaling With Composer’s Airflow Scheduler on GKE](https://medium.com/swlh/the-smarter-way-of-scaling-with-composers-airflow-scheduler-on-gke-88619238c77b) - [Roy Berkowitz](https://www.linkedin.com/in/roy-berkowitz-19922aa9/) discusses more effective use of nodes in the Cloud Composer service.
- [Better together: orchestrating your Data Fusion pipelines with Cloud Composer](https://cloud.google.com/blog/products/data-analytics/easier-management-for-cloud-etl-elt-pipelines) - [Rachael Deacon-Smith](https://www.linkedin.com/in/rachael-deacon-smith-82660172) provides an overview of the operator for Datafusion use case on Cloud Composer.


Question: How to: Non-English resources

Answer:
- [Airflow Documentation-Chinese](https://airflow.apachecn.org) - (🇨🇳Chinese) [Apachecn](https://github.com/apachecn) has translated the Airflow official documentation.
- [Gestion de Tâches avec Apache Airflow](http://ncrocfer.github.io/posts/gestion-de-taches-avec-apache-airflow/) - (🇫🇷French) [Nicolas Crocfer](https://github.com/ncrocfer) - Overview of Airflow, basic concepts and how to write and trigger a DAG.
- [Airflowはすごいぞ！100行未満で本格的なデータパイプライン](https://qiita.com/hankehly/items/1f02a34740276d1b8f0f) - (🇯🇵Japanese) [Hank Ehly](https://github.com/hankehly) gives a comprehensive introduction to Airflow's main concepts, and demonstrates how to create a data pipeline in less than 100 lines of code.
- [apache airflow 複数worker構成のalpine版docker imageを作った](https://sekailab.com/wp/2018/04/05/apache-airflow-multinode-alpine-docker-image/) - (🇯🇵Japanese) [Akio Ohta](https://github.com/Drunkar) walks through his [Docker image](https://hub.docker.com/r/drunkar/airflow-alpine/) for deploying an Alpine-based Airflow system.
- [AirflowのタスクログをS3に保存する方法](https://qiita.com/hankehly/items/5e2493cdf2c95ae42449) - (🇯🇵Japanese) [Hank Ehly](https://github.com/hankehly) shows step-by-step how to configure sending task logs to AWS S3.
- [【徹底解説】Airflow Fluentd Elasticsearch Docker の連携方法](https://qiita.com/hankehly/items/f525465bba8b47da0b76) - (🇯🇵Japanese) [Hank Ehly](https://github.com/hankehly) describes how to handle worker task logs with Fluentd, Elasticsearch and Docker.
- [Apache Airflow – Kaikki Mitä Meillä On, Lähtee Dageista](https://www.solita.fi/blogit/apache-airflow-kaikki-mita-meilla-on-lahtee-dageista/) - (🇫🇮Finnish) [Olli Iivonen](https://www.linkedin.com/in/oiivonen/)'s overview of Airflow, concepts and Airflow's usage at [Solita](https://www.solita.fi/).
- [Airflow - Automatizando seu fluxo de trabalho](https://speakerdeck.com/gilsondev/airflow-automatizando-seu-fluxo-de-trabalho) - (🇧🇷Portuguese) [Gilson Filho](https://github.com/gilsondev)'s overview of Airflow, concept and basic use.
- [Panduan Dasar Apache Airflow](https://imamdigmi.github.io/post/tutorial-airflow-part-1/) - (🇮🇩Indonesian) [Imam Digmi](https://github.com/imamdigmi) - Overview of Airflow, concept, basic use with use case.
- [Airflow](https://blog.duyet.net/tag/airflow) - (🇻🇳Vietnamese) [Duyet Le](https://github.com/duyet) - Overview of Airflow, concept, basic use with use case.
- [Michael Yang's Airflow Chinese Blog Posts](https://blog.csdn.net/Young2018/article/details/109105370?spm=1001.2014.3001.5501) - Michael Yang's Chinese blog posts about data engineering with Apache Airflow, conclude basic tutorials and devops skills.


Question: How to: Sample projects

Answer:
- [Google Cloud Platform Public Datasets Pipelines](https://github.com/GoogleCloudPlatform/public-datasets-pipelines) - Cloud-native, data pipeline architecture for onboarding datasets to the Google Cloud Public Datasets Program.
- [GitLab Data Team DAGs](https://gitlab.com/gitlab-data/analytics/-/tree/master/dags) - Several DAGs used to build analytics for the GitLab platform.
- [deploy-airflow-on-ecs-fargate](https://github.com/hankehly/deploy-airflow-on-ecs-fargate) - Deploy to Amazon ECS Fargate. Demonstrates various features and configurations, such as autoscaling workers to zero, S3 remote logging and secret management.


Question: Are there plans or ideas floating around for a logo contest, or is that a Public-Beta stage thing?

Answer:
Getting a logo is part of getting a design, which is something that happens when a site
graduates
out of public beta.
In short, not for a long time yet.
While the site is in beta it will keep this theme that it currently has.


Question:
Is it realistic or thought neccesary to develop a plugin/feature for this site, to automatically add a small info-box about a thingiverse link? An example of this is Apple's 3D touch technology. In theory, the plugin could recogize thingiverse links in questions and answers, replace the link with an image and the author/name of the project.

I'm also volunteering myself to help with this if there's interest. (Experience with Thingiverse API)

Answer:
Typically, it's a better idea to wait before you try to get this kind of thing integrated.
Enthusiasm is great in a private beta, but for the early stages, direct that enthusiasm towards the Q&A. That's what'll get this site on its feet and into a successful public beta.
When the site's more stable and running nicely, then if there's a need (or want) for a plugin like this then the discussion about it can be had.
(On a tangent - if such a plugin is going to happen, it may well be down to SE's developers to get it done, which might make getting assistance from the people on this site difficult.)


Question: What would be a good tag to use for doing business/making profit with a 3D printer?

Answer:
I'd suggest two different tags:
[monetization]: For selling, or profiting from printing, or from printing machines
[financing] (or [costs]): For calculating the cost of materials and machines, including operational expenses.


Question:
We currently have quite a few tags about filament:

filament

plastic-filament

thermoplastic-filament

etc.

I feel like we need to clean these up and make clear what we'll use each tag for.

Thoughts?

Answer:
I see a form of hierarchy that could be used.
Depending on the question a higher level tag could be used or a more specific one for specific problems (or both tags even).
Filament
Plastic Filament
ABS
PLA
Water Soluble
PVA
Flexible
NinjaFlex and similar
TPU
Conductive
Metalic
Resin
Powder
Full Color
Feel free to edit to add more types


Question:
We have only had activity on 3 questions in the past 2 days.  It is recommended that we get 10 new questions per day.  This is far below these goals.

What can we do to get more questions?

Answer:
I wouldn't worry too much about it.
We're still currently in the private beta stage, and we'll be in this spot for another 9 days. After nearly two weeks of private beta, it's perfectly natural for activity to slow down.
So what should you do?
Don't lose out. Make sure that you keep your commitment to this strong. Keep active on meta, working through the queues and discussing site issues.
Quantity is not everything. Whatever you do, never sacrifice quality in these critical stages. Don't ask questions simply for the sake of asking: if you do, you're bailing out on the site. Continue answering questions with quality. If you start seeding the site with lower-quality questions, you'll destroy the site. I've seen this happen. You've got a reputation so far, make sure to keep it, or even improve it.


Question:
As you may or may not know, this is the third iteration of a proposal site that covers 3D Printing. The first 2 made it to the beta phase, but did not graduate  from the beta successfully:

Digital Fabrication

Personal Manufacturing

Would it be acceptable to extract good/relevant questions out of these beta site question dumps and post them in the 3D Printing site?

Answer:
If someone has a question from one of those older sites, they should go ahead and ask it. But a wholesale importing of content from elsewhere is not really a desirable way to build this site.
There is a lot of ownership and careful curation that goes with vetting the content of this site. Questions imported from elsewhere would always have that air of odd, forgotten legacy content back-dated and
anonymous
with no owners or real-time vetting at all. If someone posts another answer or asks for some followup to one of these questions, no one will receive the notification. Essentially, we would be loading this site up with a lot of questions asked and answered a long time ago without imparting any of the benefits of reputation, ownership, or experience into the community that is supposed to take care of it.
That's why we don't do it.


Question: I feel like questions along the lines of, "my printer is crashing for no obvious reason, what should I do?" may be too broad and open-ended for this format. It's better handled by a forum where people can have running discussions to rule out a series of tests. What do you guys think?

Answer:
For common problems that get asked a lot, I wouldn't just close these as
too broad.
A better solution is to create a
canonical post
like this:
How do I troubleshoot when I have no clue where to start?
These attract a
lot
of users.
The goal is to create a step-by-step trouble-shooting guide to explain what lights, nozzles, and sneedles to look when you're kwigger isn't going
zong.
And don't just answer with a hyperlink to some other discussion group somewhere. Do everything you can to really overkill it. Write a detailed, step-by-step, ultra-clear guide, so when zillions of people with this problem go searching, you stand a good chance of the best possible answer on the web.
This is one of those opportunities to attract some great new users who will add value for years to come.


Question:
Methods for smoothing 3D objects

So... maybe one day it becomes community wiki. I think that we need this type of questions, so we can provide more detailed answers to methods and practices.

Answer:
"is still too broad?"
is answered by your own question,
"we can provide more detailed answers"
.
The problem essentially is that this Q&A format isn't suited to very long treatises on this type of subject.  It's far better to ask a specific question such as
"How do I smooth this ABS print to eliminate all signs that it was 3D printed?"
and receive several good answers, than to have a one-stop-wiki question that attempts (and usually does very poorly) at holding all the answers to all the possible smoothing questions.
So I'd recommend we leave this question closed and let people start more specific questions as they run into actual problems.


Question:
I've been seeing feedback from people in other 3DP communities that they think this stack exchange site is driving away new users by holding newbie questions to an excessively high standard for quality. On one hand, we all know SE works best with clear and logical questions that lead to clear and logical answers. But the majority of people seeking help with 3d printers don't seem to have enough of a technical foundation to know how to ask good questions. There's a large potential userbase (perhaps MOST potential users) that will need handholding for their first few questions.

How do you guys want to handle this?

Answer:
I think you highlighted one of the more important points, in that "
SE works best with clear and logical questions that lead to clear and logical answers
". From what I've noticed (and I just went back through my own voting history), there have been a number of "primarily opinion based" and "too broad" questions. I believe it's important to maintain quality questions/answers especially in this early stage of release. Please regard
this other meta post
asking what the guidelines are for proper 3D Printing community questions.
I, for one, feel that I learned the most SE etiquette by reading a large number of questions on SO as opposed to the SE documentation. It seems that a few of the questions we've gotten lately have been from completely new users to the Stack Exchange network. I don't like scaring people away from the site, so it is best to try and coach these new users.
I retract the following suggestion as I agree with
Mark Booth's answer
given his explanation.
I would suggest an informal guideline for closing questions:
First, notify the OP to the condition of their question. Perhaps even suggest a means to fix the errant condition(s).
If, after at least 24 hours of the comment, the OP has not either responded reasonably (within SE etiquette) nor updated the question, then begin the process of closing.
I'll leave this open to the community for amendments below:


Question:
I just noticed that there is a large amount of tag wiki excerpts edits in the review queue, all of which are of the form:

X is for questions about X

A tag wiki excerpt should define what a term means to our community specifically and give usage advice. In particular, Stack Exchange offers the following default reason for rejecting an excerpt:

Tag excerpts amounting to, "[tag] is for questions about [tag]" are pointless and usually rejected. Excerpts should describe why and when a tag would be used.

See the help center for more details on what a tag excerpt should be.

While obviously well-intended, I believe such tag wiki excerpts should not be suggested (and/or approved). This post is to serve as a gentle reminder of that.

Answer:
For reference, I would like to propose a copy-paste solution for tags.
Usage Guidance
```
For questions regarding {insert list of applicable topics} of {Tag Name}.
```
Details
```
```
{Tag name, unabbreviated}: {Definition}

Examples:
 - What is {blah blah blah}?
 - Where can I find {blah blah blah}?
 - Why does {blah blah blah}?
 - How do I {blah blah blah}?
```
```


Question:
The amount of posts being voted to closed is getting ridiculous.  The last two posts have been printing related, one looking for information and the other a design question for 3D printing yet both have been voted to be closed.

Yes, I have read the other Meta post about how closing is not permanent (Closing/locking too many questions?) but this site does not have much to offer right now and I can tell you if I was new here I wouldn't stick around and edit the post time after time to get an answer when I'm sure most people here can answer the question.  The site is now running at 1.2 questions per day and I don't see that going up at all if the criteria isn't changed for how people are voting.  If that's how everyone wants the site run then that's fine but I'm sure you'll be alone here.

Answer:
I absolutely agree. I believe we all want this site to maintain high quality, but right now almost no questions fall within our desired scope and form.
I think we either need to:
Change the acceptable scope of questions to be asked
Change how we welcome new users
Right now most new users do not ask questions "the SE way", which quickly leads to down-votes and closing votes. For new users this is a direct slap in the face.
What we rather should do is to
encourage
new users to improve their question, and if they do, give them the highly desired up-votes to make them come back for more.
I think the reputation system on SE sites is a great motivator for writing good questions and answers. And if we want this site to grow, we need to let our fellow users grow with it.


Question:
There has been a lot of questions and confusion about what is acceptable here on 3D Printing SE. Let's go ahead and discuss what types of questions should be acceptable on the site.

We're going to do this based on votes. I'll leave it up to debate in the comments below, but we'll emulate Area51 question requirements. Here are the guidelines:

10 positive votes, passes.

One answer per topic. The focus is voting on topics, and votes pertain to the validity of the topic.

Similar topics that can be merged should explicitly mention the deprecated topic.

Appropriately passed topics should have closed questions re-opened (as appropriate).

Answer:
I absolutely agree. I believe we all want this site to maintain high quality, but right now almost no questions fall within our desired scope and form.
I think we either need to:
Change the acceptable scope of questions to be asked
Change how we welcome new users
Right now most new users do not ask questions "the SE way", which quickly leads to down-votes and closing votes. For new users this is a direct slap in the face.
What we rather should do is to
encourage
new users to improve their question, and if they do, give them the highly desired up-votes to make them come back for more.
I think the reputation system on SE sites is a great motivator for writing good questions and answers. And if we want this site to grow, we need to let our fellow users grow with it.


Question:
Context: I have absolutely no knowledge of 3D printing other than you need a computer, a printer, some software, and a design. That is literally the extent of my knowledge on 3D printing.

However, I have an idea of something I'd like to have 3D printed. While the idea has a fairly reasonably defined shape in mind, it doesn't exist in any digital or paper format and has some specifics that still need to be filled out (such as accurate dimensions and a few design details).

I need to bridge the chasm of knowledge between my current design and limited knowledge to a fleshed-out design file with chosen materials and other specifics. A good chunk of my problem is that I don't even know what I should know. While I realize that this site is still fairly new and things are being nailed down, how do I ask the proper question(s) to fill in my knowledge gaps that will be on-topic for the site? Is there even a path forward for these kinds of questions on the site?

Answer:
There is a
new question on Meta
that should help define what is okay on this site.
However, your question is important to address here.
Ultimately, you shouldn't be afraid to go ahead and ask the question. If the question does not meet the conditions of the present site, the general public will be sure to let you know and (hopefully) help direct you in at least asking a more direct question.
If you post a question that is closed, it would acceptable to post a question here on Meta that specifically asks how to make your question fit within the scope of the site.
I would suggest providing as much information as you have and feel free to ask the more general questions about 3D printing. Most people in the community will ask specific questions to try and help you. Some may even be able to fill in the blanks of what you're asking and provide you with very helpful answers.


Question:
We have a recent question that brings up the question of "Should we support general hobbyist questions?"

Currently, there doesn't appear to be a viable site within the SE network. The question at hand seems to be a mix between 3D Printing and DIY. If we allow this question, it could allow people to ask questions like the following:

CNC Mills

Routers

Lasers

etc.

Answer:
There is a
new question on Meta
that should help define what is okay on this site.
However, your question is important to address here.
Ultimately, you shouldn't be afraid to go ahead and ask the question. If the question does not meet the conditions of the present site, the general public will be sure to let you know and (hopefully) help direct you in at least asking a more direct question.
If you post a question that is closed, it would acceptable to post a question here on Meta that specifically asks how to make your question fit within the scope of the site.
I would suggest providing as much information as you have and feel free to ask the more general questions about 3D printing. Most people in the community will ask specific questions to try and help you. Some may even be able to fill in the blanks of what you're asking and provide you with very helpful answers.


Question: It's pretty manageable right now due to the low question rate, but I think maybe ~3% of all questions this site will get, forever, will be "what's the best printer" or "what printer should I buy" type questions. They're mostly coming from people who don't know enough about 3DP to articulate their requirements, so they're difficult to help and very unfocused. Is there a better way to handle this than locking them as they come up?

Answer:
I think the site tries to prevent this as best as possible, short of having a bot prevent the post.
There is the
Don't Ask
page and the
On-Topic
(which we have control over its content).
The suggested/related questions that appear when asking a question. Ideally users at least regard its existence before posting the question.
There appears to be a pop-up that does try to warn the user that their question is "subjective". I tested it really quick, see the image below
All else fails, that's what moderators are for. Moderators should utilize the tools available to them. So even if these types of questions pile up, there are means of quickly identifying. A simple feature, available to everyone, is advanced searches. I try to use searches like
this
to quickly identify large groups of questions that may need attention.
It might be nice to have more options over whether keywords can be contained or not, "asker" reputation, "answerer" reputation, etc.


Question:
This question was migrated to electronics.SE. To me the migration makes no sense:

The question deals with a problem that is relevant to the users of this site

This site can provide a more specialized answer than electronics.SE can provide: you don't just need to know whether it's a 10k or 100k thermistor, but also figure out the correct thermistor table in your firmware. electronics.SE does not have knowledge of 3D printer firmware, which is the issue underlying this question.

Obviously there's some overlap between Arduino/Electronics/3D printing, but what determines whether a question should be migrated?

Answer:
The question was migrated because the specific question of "
How do I tell if my Thermistors are 10k or 100k?
" is going to be best answered by users of Electrical Engineering SE. This also provides the SE network with more appropriate traffic based on the question at hand.
However, if the question of "
How can I change the thermistors settings in Marlin firmware?
" were to arise, then the question would be best suited here on 3D Printing. It might help both SE sites by providing links to each other's relevant questions for future users to reference.
If the question was something like "
How can I wire a hotend?
", this would be more appropriate here on 3D Printing SE as users in Electrical Engineering SE may not know as much about the topic compared to users in 3D Printing. This may be a poor example, but the idea is that there is strict correlation between
hotends
and 3D printing, whereas identifying thermistors is not a specific topic to just 3D printing.
Update
After reading a few posts on SE meta,
this one
leads me to agree with you that this particular question may not have needed to be migrated. However, it exposes an important question of how we want to proceed with questions like this in the future? How far down the rabbit hole do we want to allow this site to go in this topic? I'd recommend others pitch in recommendations in answers here on what would be the appropriate topic in this case that can be applied to our
On-Topic
page so that it may be amended.


Question:
Following this issue I would like to propose to allow general CNC related questions here as well. 3D Printers are CNC machines, they only add instead of subtract. 3D printers use G-Code, just like CNC milling machines.

3D Printing alone makes for a very narrow community if it excludes subtractive manufacturing questions. For example, there is no python.stackexchange.com or javascript.stackexchange.com, all of that goes into one network: stackoverflow.com. That's just the same level of division.

Navel-gazing has never done anyone good :)

Answer:
As has been discussed previously
here
, the relevance of CNC questions on this site depend on the questions closeness to 3D printing. So, for instance, if the software you are asking about is the same as a 3D printer would use, I would agree that it can be asked here. However, if the software is dedicated to CNC machines and unusable for 3D printers, it does not belong here at the time being.
If the software you are looking for indeed works for both 3D printers and CNC machines, then perhaps you could specify this in your question to make it more relevant to this site?


Question: I know that here at SE I cant ask recommendation of products and so. I have a printer that I wanna buy and I want to know if it's good. Where should I post this question in order to get some smart people to take a look?

Answer:
Great question! 3DPrinting SE attracts a lot of new Stack Exchange network users, which unfortunately can come with questions that don't always meet the Q&A style for the network that this site tries to uphold.
Questions like this are probably best asked in the
Chat
room. Currently, it's not very active, but hopefully if we have a few people interested in these more "off-topic" questions the activity will pick up. Pings are going to be a very useful tool if we're going to try and utilize the chat window more. Pinging some of the highly active users may help get quality answers to those off-topic questions.


Question:
Do we want to add a BLOG to this site?
Blogs are another way of communicating things that don't fit the Q&A model.

Here is a good description of what a Blog is and how you can start one.

This post on the SuperUser Blog asking for help with their Blog is helpful too.

Answer:
+1 - As it seems a good idea. Not sure what would go in it though - do you have any concrete examples of blog ideas?
I already have a (messy)
blog
, and I am not sure if I could also write a blog on here too. I wonder what the score is regarding duplicating personal blogs on to SE blogs?
Content
An SE 3D print blog could be a good idea for
Build Logs
, for example, maybe. Or maybe a page that links to other people's superlative build logs,
David Crocker's blogs
come to mind.
Is the blog (unlike the SE Q&A site) allowed to contain
links to cheap items/suppliers
? Apparently, reviews are allowed, from
Blog Overflow
:
Review a product
. Reviews don’t fit the Q&A nature of the sites, but these rules don’t apply on the blog!  Between a review written by a random person on the internet and a review written by a user on the site who consistently gets a lot of upvotes, which review would you trust more?
This closed question would have fitted into a blog nicely:
What Is 3D Printing?
Concerns
So, yes, I think it would be worth starting one up and seeing how it goes... although I would be a little concerned about the regularity of posts:
Plan a schedule
. Given the results of steps #2 and #3, think about a rough idea of a schedule for the blog. Will there be one post a week, posted Mondays? Will there be posts on Tuesdays and posts on Fridays? You don’t need to be pushing out posts daily, but you should post at least once a week.
and
Pick a posting schedule and stick to it
. It is easier to simply keep up from the get go than catch up if you fall behind.  Have a couple draft posts stashed away for a rainy day, ready to go that can be published if there is a lull.
Also, who would coordinate it? Are you putting yourself forward?
Have someone holding the reins
. This person doesn’t need to be the one writing all the posts, just someone that helps coordinate who is writing what and when it is getting posted.
However, it
is
a bit of a misfortune that Blog Overflow has a rather unfortunately acronym (as well as sounding like
Bog
Overflow).


Question: Should we add a tag for 'laser' for those of us who are attaching laser cutters/burners to 3D printers to turn them into 3D CNC. Or is this drifting too far off topic from "3d printing"?

Answer:
A good question. I'm currently using insights I obtained from mucking about with a 3D printer with Arduino Mega 2560 and RAMPS 1.4 to dive into retrofitting an elderly CNC router (but, as a router, still, not as a printer) with a new control system (presently thinking and have on order Arduino Uno R3 and Arduino CNC shield R3, we'll see how implementation goes on those) and was wondering if I could ask questions here that would overlap, or not, as I don't think there's a CNC/subtractive community similar to this 3DP/additive one in the ecosystem yet.
I actually started out thinking I could just use the same controller setup (there's plenty of stepper driver slots available, and going "dual-purpose" cut and print is potentially interesting) but it seems the official GRBL fork is not 2560 compatible, though there is a fork to try and make it so - it seemed more of a sure bet to stick with the "official" fork and the common hardware.
Likewise I've seen a bit of chatter about making Marlin switch-hit CNC/3DP, but I get the impression it's not all there yet, and I'm more interested in immediately usable based on current/past efforts than trying to develop new functionality my dang self.