Question: What is continuous delivery?

Answer:
Continuous delivery (CD) means frequently delivering updates to your software application.
The idea is that by updating more often, you do not have to wait for a specific timed period, and your organization
gets better at the process of responding to change.
Some Ansible users are deploying updates to their end users on an hourly or even more frequent basis – sometimes every time
there is an approved code change.  To achieve this, you need tools to be able to quickly apply those updates in a zero-downtime way.
This document describes in detail how to achieve this goal, using one of Ansible’s most complete example
playbooks as a template: lamp_haproxy. This example uses a lot of Ansible features: roles, templates,
and group variables, and it also comes with an orchestration playbook that can do zero-downtime
rolling upgrades of the web application stack.
The playbooks deploy Apache, PHP, MySQL, Nagios, and HAProxy to a CentOS-based set of servers.
We’re not going to cover how to run these playbooks here. Read the included README in the GitHub project along with the
example for that information. Instead, we’re going to take a close look at every part of the playbook and describe what it does.


Question: Reusable content: roles

Answer:
By now you should have a bit of understanding about roles and how they work in Ansible. Roles are a way to organize
content: tasks, handlers, templates, and files, into reusable components.
This example has six roles:common,base-apache,db,haproxy,nagios, andweb. How you organize
your roles is up to you and your application, but most sites will have one or more common roles that are applied to
all systems, and then a series of application-specific roles that install and configure particular parts of the site.
Roles can have variables and dependencies, and you can pass in parameters to roles to modify their behavior.
You can read more about roles in theRolessection.


Question: Configuration: group variables

Answer:
Group variables are variables that are applied to groups of servers. They can be used in templates and in
playbooks to customize behavior and to provide easily-changed settings and parameters. They are stored in
a directory calledgroup_varsin the same location as your inventory.
Here is lamp_haproxy’sgroup_vars/allfile. As you might expect, these variables are applied to all of the machines in your inventory:
This is a YAML file, and you can create lists and dictionaries for more complex variable structures.
In this case, we are just setting two variables, one for the port for the web server, and one for the
NTP server that our machines should use for time synchronization.
Here’s another group variables file. This isgroup_vars/dbserverswhich applies to the hosts in thedbserversgroup:
If you look in the example, there are group variables for thewebserversgroup and thelbserversgroup, similarly.
These variables are used in a variety of places. You can use them in playbooks, like this, inroles/db/tasks/main.yml:
You can also use these variables in templates, like this, inroles/common/templates/ntp.conf.j2:


Question: The rolling upgrade

Answer:
Now you have a fully-deployed site with web servers, a load balancer, and monitoring. How do you update it? This is where Ansible’s
orchestration features come into play. While some applications use the term ‘orchestration’ to mean basic ordering or command-blasting, Ansible
refers to orchestration as ‘conducting machines like an orchestra’, and has a pretty sophisticated engine for it.
Ansible has the capability to do operations on multi-tier applications in a coordinated way, making it easy to orchestrate a sophisticated zero-downtime rolling upgrade of our web application. This is implemented in a separate playbook, calledrolling_update.yml.
Looking at the playbook, you can see it is made up of two plays. The first play is very simple and looks like this:
What’s going on here, and why are there no tasks? You might know that Ansible gathers “facts” from the servers before operating upon them. These facts are useful for all sorts of things: networking information, OS/distribution versions, and so on. In our case, we need to know something about all of the monitoring servers in our environment before we perform the update, so this simple play forces a fact-gathering step on our monitoring servers. You will see this pattern sometimes, and it is a useful trick to know.
The next part is the update play. The first part looks like this:
This is just a normal play definition, operating on thewebserversgroup. Theserialkeyword tells Ansible how many servers to operate on at once. If it is not specified, Ansible will parallelize these operations up to the default “forks” limit specified in the configuration file. But for a zero-downtime rolling upgrade, you may not want to operate on that many hosts at once. If you had just a handful of webservers, you may want to setserialto 1, for one host at a time. If you have 100, maybe you could setserialto 10, for ten at a time.


Question: Managing other load balancers

Answer:
In this example, we use the simple HAProxy load balancer to front-end the web servers. It is easy to configure and easy to manage. As we have mentioned, Ansible has support for a variety of other load balancers like Citrix NetScaler, F5 BigIP, Amazon Elastic Load Balancers, and more.
For other load balancers, you may need to send shell commands to them (like we do for HAProxy above), or call an API, if your load balancer exposes one. For the load balancers for which Ansible has modules, you may want to run them as alocal_actionif they contact an API. You can read more about local actions in theControlling where tasks run: delegation and local actionssection.  Should you develop anything interesting for some hardware where there is not a module, it might make for a good contribution!


Question: Continuous delivery end-to-end

Answer:
Now that you have an automated way to deploy updates to your application, how do you tie it all together? A lot of organizations use a continuous integration tool likeJenkinsorAtlassian Bambooto tie the development, test, release, and deploy steps together. You may also want to use a tool likeGerritto add a code review step to commits to either the application code itself, or to your Ansible playbooks, or both.
Depending on your environment, you might be deploying continuously to a test environment, running an integration test battery against that environment, and then deploying automatically into production.  Or you could keep it simple and just use the rolling-update for on-demand deployment into test or production specifically.  This is all up to you.
For integration with Continuous Integration systems, you can easily trigger playbook runs using theansible-playbookcommand line tool, or, if you’re using AWX, thetower-clicommand or the built-in REST API.  (The tower-cli command ‘joblaunch’ will spawn a remote job over the REST API and is pretty slick).
This should give you a good idea of how to structure a multi-tier application with Ansible and orchestrate operations upon that app, with the eventual goal of continuous delivery to your customers. You could extend the idea of the rolling upgrade to several different parts of the app; maybe add front-end web servers along with application servers, or replace the SQL database with NoSQL database. Ansible gives you the capability to easily manage complicated environments and automate common operations.


Question: Enforcing or preventing check mode on tasks

Answer:
If you want certain tasks to run in check mode always, or never, regardless of whether you run the playbook with or without--check, you can add thecheck_modeoption to those tasks:
For example:
Running single tasks withcheck_mode:truecan be useful for testing Ansible modules, either to test the module itself or to test the conditions under which a module would make changes. You can register variables (seeConditionals) on these tasks for even more detail on the potential changes.


Question: Skipping tasks or ignoring errors in check mode

Answer: If you want to skip a task or ignore errors on a task when you run Ansible in check mode, you can use a boolean magic variableansible_check_mode, which is set toTruewhen Ansible runs in check mode. For example:


Question: Enforcing or preventing diff mode on tasks

Answer: Because the--diffoption can reveal sensitive information, you can disable it for a task by specifyingdiff:false. For example:


Question: Become connection variables

Answer:
You can define differentbecomeoptions for each managed node or group. You can define these variables in inventory or use them as normal variables.
For example, if you want to run all tasks asrooton a server namedwebserver, but you can only connect as themanageruser, you could use an inventory entry like this:


Question: Risks and limitations of become

Answer:
Although privilege escalation is mostly intuitive, there are a few limitations
on how it works.  Users should be aware of these to avoid surprises.


Question: Risks of becoming an unprivileged user

Answer:
Ansible modules are executed on the remote machine by first substituting the
parameters into the module file, then copying the file to the remote machine,
and finally executing it there.
Everything is fine if the module file is executed without usingbecome,
when thebecome_useris root, or when the connection to the remote machine
is made as root. In these cases, Ansible creates the module file with
permissions that only allow reading by the user and root, or only allow reading
by the unprivileged user being switched to.
However, when both the connection user and thebecome_userare unprivileged,
the module file is written as the user that Ansible connects as (theremote_user), but the file needs to be readable by the user Ansible is set
tobecome. The details of how Ansible solves this can vary based on the platform.
However, on POSIX systems, Ansible solves this problem in the following way:
First, ifsetfaclis installed and available in the remotePATH,
and the temporary directory on the remote host is mounted with POSIX.1e
filesystem ACL support, Ansible will use POSIX ACLs to share the module file
with the second unprivileged user.
Next, if POSIX ACLs arenotavailable orsetfaclcould not be
run, Ansible will attempt to change ownership of the module file usingchownfor systems that support doing so as an unprivileged user.
New in Ansible 2.11, at this point, Ansible will trychmod +awhich
is a macOS-specific way of setting ACLs on files.


Question: Not supported by all connection plugins

Answer:
Privilege escalation methods must also be supported by the connection plugin
used. Most connection plugins will warn if they do not support become. Some
will just ignore it as they always run as root (jail, chroot, and so on).


Question: Only one method may be enabled per host

Answer:
Methods cannot be chained. You cannot usesudo/bin/su-to become a user,
you need to have privileges to run the command as that user in sudo or be able
tosudirectly to it (the same forpbrun,pfexecor other supported methods).


Question: Privilege escalation must be general

Answer:
You cannot limit privilege escalation permissions to certain commands.
Ansible does not always
use a specific command to do something but runs modules (code) from
a temporary file name which changes every time.  If you have ‘/sbin/service’
or ‘/bin/chmod’ as the allowed commands this will fail with Ansible as those
paths won’t match with the temporary file that Ansible creates to run the
module. If you have security rules that constrain yoursudo/pbrun/doasenvironment
to run specific command paths only, use Ansible from a special account that
does not have this constraint, or use AWX or theRed Hat Ansible Automation Platformto manage indirect access to SSH credentials.


Question: May not access environment variables populated by pamd_systemd

Answer:
For most Linux distributions usingsystemdas their init, the default
methods used bybecomedo not open a new “session”, in the sense ofsystemd. Because thepam_systemdmodule will not fully initialize a new
session, you might have surprises compared to a normal session opened through
ssh: some environment variables set bypam_systemd, most notablyXDG_RUNTIME_DIR, are not populated for the new user and instead inherited
or just emptied.
This might cause trouble when trying to invokesystemdcommands that depend onXDG_RUNTIME_DIRto access the bus:
To forcebecometo open a newsystemdsession that goes throughpam_systemd, you can usebecome_method:machinectl.
For more information, seethis systemd issue.


Question: Resolving Temporary File Error Messages

Answer: Failed to set permissions on the temporary files Ansible needs to create when becoming an unprivileged user”This error can be resolved by installing the package that provides thesetfaclcommand. (This is frequently theaclpackage but check your OS documentation.)


Question: Become and network automation

Answer:
As of version 2.6, Ansible supportsbecomefor privilege escalation (enteringenablemode or privileged EXEC mode) on all Ansible-maintained network platforms that supportenablemode. Usingbecomereplaces theauthorizeandauth_passoptions in aproviderdictionary.
You must set the connection type to eitherconnection:ansible.netcommon.network_cliorconnection:ansible.netcommon.httpapito usebecomefor privilege escalation on network devices. Check thePlatform Optionsdocumentation for details.
You can use escalated privileges on only the specific tasks that need them, on an entire play, or on all plays. Addingbecome:trueandbecome_method:enableinstructs Ansible to enterenablemode before executing the task, play, or playbook where those parameters are set.
If you see this error message, the task that generated it requiresenablemode to succeed:
To setenablemode for a specific task, addbecomeat the task level:
To set enable mode for all tasks in a single play, addbecomeat the play level:


Question: Setting enable mode for all tasks

Answer:
Often you wish for all tasks in all plays to run using privilege mode, which is best achieved by usinggroup_vars:
group_vars/eos.yml


Question: authorize and auth_pass

Answer:
Ansible still supportsenablemode withconnection:localfor legacy network playbooks. To enterenablemode withconnection:local, use the module optionsauthorizeandauth_pass:
We recommend updating your playbooks to usebecomefor network-deviceenablemode consistently. The use ofauthorizeandproviderdictionaries will be deprecated in the future. Check thePlatform Optionsdocumentation for details.


Question: Administrative rights

Answer:
Many tasks in Windows require administrative privileges to complete. When using
therunasbecome method, Ansible will attempt to run the module with the
full privileges that are available to the become user. If it fails to elevate
the user token, it will continue to use the limited token during execution.
A user must have theSeDebugPrivilegeto run a become process with elevated
privileges. This privilege is assigned to Administrators by default. If the
debug privilege is not available, the become process will run with a limited
set of privileges and groups.
To determine the type of token that Ansible was able to get, run the following
task:
The output will look something similar to the below:
Under thelabelkey, theaccount_nameentry determines whether the user
has Administrative rights. Here are the labels that can be returned and what
they represent:
Medium: Ansible failed to get an elevated token and ran under a limited
token. Only a subset of the privileges assigned to the user are available during
the module execution and the user does not have administrative rights.High: An elevated token was used and all the privileges assigned to the
user are available during the module execution.System: TheNTAUTHORITY\Systemaccount is used and has the highest
level of privileges available.


Question: Local service accounts

Answer:
Prior to Ansible version 2.5,becomeonly worked on Windows with a local or domain
user account. Local service accounts likeSystemorNetworkServicecould not be used asbecome_userin these older versions. This restriction
has been lifted since the 2.5 release of Ansible. The three service accounts
that can be set underbecome_userare:
SystemNetworkServiceLocalService
Because local service accounts do not have passwords, theansible_become_passwordparameter is not required and is ignored if
specified.


Question: Become without setting a password

Answer:
As of Ansible 2.8,becomecan be used to become a Windows local or domain account
without requiring a password for that account. For this method to work, the
following requirements must be met:
The connection user has theSeDebugPrivilegeprivilege assignedThe connection user is part of theBUILTIN\AdministratorsgroupThebecome_userhas either theSeBatchLogonRightorSeNetworkLogonRightuser right
Using become without a password is achieved in one of two different methods:
Duplicating an existing logon session’s token if the account is already logged onUsing S4U to generate a logon token that is valid on the remote host only
In the first scenario, the become process is spawned from another logon of that
user account. This could be an existing RDP logon, console logon, but this is
not guaranteed to occur all the time. This is similar to theRunonlywhenuserisloggedonoption for a Scheduled Task.
In the case where another logon of the become account does not exist, S4U is
used to create a new logon and run the module through that. This is similar to
theRunwhetheruserisloggedonornotwith theDonotstorepasswordoption for a Scheduled Task. In this scenario, the become process will not be
able to access any network resources like a normal WinRM process.


Question: Accounts without a password

Answer:
Ansible can be used to become a Windows account that does not have a password (like theGuestaccount). To become an account without a password, set up the
variables like normal but setansible_become_password:''.
Before become can work on an account like this, the local policyAccounts: Limit local account use of blank passwords to console logon onlymust be disabled. This can either be done through a Group Policy Object (GPO)
or with this Ansible task:


Question: Become flags for Windows

Answer:
Ansible 2.5 added thebecome_flagsparameter to therunasbecome method.
This parameter can be set using thebecome_flagstask directive or set in
Ansible’s configuration usingansible_become_flags. The two valid values
that are initially supported for this parameter arelogon_typeandlogon_flags.
The keylogon_typesets the type of logon operation to perform. The value
can be set to one of the following:
interactive: The default logon type. The process will be run under a
context that is the same as when running a process locally. This bypasses all
WinRM restrictions and is the recommended method to use.batch: Runs the process under a batch context that is similar to a
scheduled task with a password set. This should bypass most WinRM
restrictions and is useful if thebecome_useris not allowed to log on
interactively.new_credentials: Runs under the same credentials as the calling user, but
outbound connections are run under the context of thebecome_userandbecome_password, similar torunas.exe/netonly. Thelogon_flagsflag should also be set tonetcredentials_only. Use this flag if
the process needs to access a network resource (like an SMB share) using a
different set of credentials.network: Runs the process under a network context without any cached
credentials. This results in the same type of logon session as running a
normal WinRM process without credential delegation and operates under the same
restrictions.network_cleartext: Like thenetworklogon type, but instead caches
the credentials so it can access network resources. This is the same type of
logon session as running a normal WinRM process with credential delegation.
For more information, seedwLogonType.
Thelogon_flagskey specifies how Windows will log the user on when creating
the new process. The value can be set to none or multiple of the following:
with_profile: The default logon flag set. The process will load the
user’s profile in theHKEY_USERSregistry key toHKEY_CURRENT_USER.netcredentials_only: The process will use the same token as the caller
but will use thebecome_userandbecome_passwordwhen accessing a remote
resource. This is useful in inter-domain scenarios where there is no trust
relationship, and should be used with thenew_credentialslogon_type.


Question: Limitations of become on Windows

Answer:
Running a task withasyncandbecomeon Windows Server 2008, 2008 R2
and Windows 7 only works when using Ansible 2.7 or newer.By default, the become user logs on with an interactive session, so it must
have the right to do so on the Windows host. If it does not inherit theSeAllowLogOnLocallyprivilege or inherits theSeDenyLogOnLocallyprivilege, the become process will fail. Either add the privilege or set thelogon_typeflag to change the logon type used.Prior to Ansible version 2.3, become only worked whenansible_winrm_transportwas eitherbasicorcredssp. This
restriction has been lifted since the 2.4 release of Ansible for all hosts
except Windows Server 2008 (non R2 version).The Secondary Logon serviceseclogonmust be running to useansible_become_method:runasThe connection user must already be an Administrator on the Windows host to
userunas. The target become user does not need to be an Administrator
though.


Question: Adding tags with the tags keyword

Answer: You can add tags to a single task or include. You can also add tags to multiple tasks by defining them at the level of a block, play, role, or import. The keywordtagsaddresses all these use cases. Thetagskeyword always defines tags and adds them to tasks; it does not select or skip tasks for execution. You can only select or skip tasks based on tags at the command line when you run a playbook. SeeSelecting or skipping tags when you run a playbookfor more details.


Question: Adding tags to individual tasks

Answer:
At the simplest level, you can apply one or more tags to an individual task. You can add tags to tasks in playbooks, in task files, or within a role. Here is an example that tags two tasks with different tags:
You can apply the same tag to more than one individual task. This example tags several tasks with the same tag, “ntp”:
If you ran these four tasks in a playbook with--tagsntp, Ansible would run the three tasks taggedntpand skip the one task that does not have that tag.


Question: Selecting or skipping tags when you run a playbook

Answer:
Once you have added tags to your tasks, includes, blocks, plays, roles, and imports, you can selectively execute or skip tasks based on their tags when you runansible-playbook. Ansible runs or skips all tasks with tags that match the tags you pass at the command line. If you have added a tag at the block or play level, withroles, or with an import, that tag applies to every task within the block, play, role, or imported role or file. If you have a role with several tags and you want to call subsets of the role at different times, eitheruse it with dynamic includes, or split the role into multiple roles.
ansible-playbookoffers five tag-related command-line options:
--tagsall- run all tasks, tagged and untagged except ifnever(default behavior).--tagstag1,tag2- run only tasks with either the tagtag1or the tagtag2(also those taggedalways).--skip-tagstag3,tag4- run all tasks except those with either the tagtag3or the tagtag4ornever.--tagstagged- run only tasks with at least one tag (neveroverrides).--tagsuntagged- run only tasks with no tags (alwaysoverrides).
For example, to run only tasks and blocks tagged eitherconfigurationorpackagesin a very long playbook:
To run all tasks except those taggedpackages:
To run all tasks, even those excluded because are taggednever:


Question: Previewing the results of using tags

Answer:
When you run a role or playbook, you might not know or remember which tasks have which tags, or which tags exist at all. Ansible offers two command-line flags foransible-playbookthat help you manage tagged playbooks:
--list-tags- generate a list of available tags--list-tasks- when used with--tagstagnameor--skip-tagstagname, generate a preview of tagged tasks
For example, if you do not know whether the tag for configuration tasks isconfigorconfin a playbook, role, or tasks file, you can display all available tags without running any tasks:
If you do not know which tasks have the tagsconfigurationandpackages, you can pass those tags and add--list-tasks. Ansible lists the tasks but does not execute any of them.
These command-line flags have one limitation: they cannot show tags or tasks within dynamically included files or roles. SeeComparing includes and imports: dynamic and static reusefor more information on differences between static imports and dynamic includes.


Question: Selectively running tagged tasks in reusable files

Answer:
If you have a role or a tasks file with tags defined at the task or block level, you can selectively run or skip those tagged tasks in a playbook if you use a dynamic include instead of a static import. You must use the same tag on the included tasks and on the include statement itself. For example, you might create a file with some tagged and some untagged tasks:
And you might include the tasks file above in a playbook:
When you run the playbook withansible-playbook-ihostsmyplaybook.yml--tags"mytag", Ansible skips the task with no tags, runs the tagged individual task, and runs the two tasks in the block. Also it could run fact gathering (implicit task) as it is tagged withalways.


Question: Tag inheritance: adding tags to multiple tasks

Answer:
If you want to apply the same tag or tags to multiple tasks without adding atagsline to every task, you can define the tags at the level of your play or block, or when you add a role or import a file. Ansible applies the tags down the dependency chain to all child tasks. With roles and imports, Ansible appends the tags set by therolessection or import to any tags set on individual tasks or blocks within the role or imported file. This is called tag inheritance. Tag inheritance is convenient because you do not have to tag every task. However, the tags still apply to the tasks individually.
With plays, blocks, therolekeyword, and static imports, Ansible applies tag inheritance, adding the tags you define to every task inside the play, block, role, or imported file. However, tag inheritance doesnotapply to dynamic reuse withinclude_roleandinclude_tasks. With dynamic reuse (includes), the tags you define apply only to the include itself. If you need tag inheritance, use a static import. If you cannot use an import because the rest of your playbook uses includes, seeTag inheritance for includes: blocks and the apply keywordfor ways to work around this behavior.
You can apply tags to dynamic includes in a playbook. As with tags on an individual task, tags on aninclude_*task apply only to the include itself, not to any tasks within the included file or role. If you addmytagto a dynamic include, then run that playbook with--tagsmytag, Ansible runs the include itself, runs any tasks within the included file or role tagged withmytag, and skips any tasks within the included file or role without that tag. SeeSelectively running tagged tasks in reusable filesfor more details.


Question: Configuring tags globally

Answer: If you run or skip certain tags by default, you can use theTAGS_RUNandTAGS_SKIPoptions in Ansible configuration to set those defaults.


Question: Enabling the debugger

Answer:
The debugger is not enabled by default. If you want to invoke the debugger during playbook execution, you must enable it first.
Use one of these three methods to enable the debugger:


Question: Enabling the debugger with thedebuggerkeyword

Answer:
You can use thedebuggerkeyword to enable (or disable) the debugger for a specific play, role, block, or task. This option is especially useful when developing or extending playbooks, plays, and roles. You can enable the debugger on new or updated tasks. If they fail, you can fix the errors efficiently. Thedebuggerkeyword accepts five values:
When you use thedebuggerkeyword, the value you specify overrides any global configuration to enable or disable the debugger. If you definedebuggerat multiple levels, such as in a role and in a task, Ansible honors the most granular definition. The definition at the play or role level applies to all blocks and tasks within that play or role unless they specify a different value. A definition at the block level overrides a definition at the play or role level and applies to all tasks within that block unless they specify a different value. A definition at the task level always applies to the task; it overrides definitions at the block, play, or role level.


Question: Enabling the debugger in configuration or an environment variable

Answer:
You can enable the task debugger globally with a setting inansible.cfgor with an environment variable. The only options areTrueorFalse. If you set the configuration option or environment variable toTrue, Ansible runs the debugger on failed tasks by default.
To enable the task debugger fromansible.cfg, add this setting to the[defaults]section:
To enable the task debugger with an environment variable, pass the variable when you run your playbook:
When you enable the debugger globally, every failed task invokes the debugger, unless the role, play, block, or task explicitly disables the debugger. If you need more granular control over what conditions trigger the debugger, use thedebuggerkeyword.


Question: Enabling the debugger as a strategy

Answer:
If you are running legacy playbooks or roles, you may see the debugger enabled as astrategy. You can do this at the play level, inansible.cfg, or with the environment variableANSIBLE_STRATEGY=debug. For example:
Or in ansible.cfg:


Question: Resolving errors in the debugger

Answer:
After Ansible invokes the debugger, you can use the sevendebugger commandsto resolve the error that Ansible encountered. Consider this example playbook, which defines thevar1variable but uses the undefinedwrong_varvariable in a task by mistake.
If you run this playbook, Ansible invokes the debugger when the task fails. From the debug prompt, you can change the module arguments or the variables and run the task again.
Changing the task arguments in the debugger to usevar1instead ofwrong_varmakes the task run successfully.


Question: Available debug commands

Answer:
You can use these seven commands at the debug prompt:
For more details, see the individual descriptions and examples below.


Question: Update args command

Answer:
task.args[*key*]=*value*updates a module argument. This sample playbook has an invalid package name.
When you run the playbook, the invalid package name triggers an error, and Ansible invokes the debugger. You can fix the package name by viewing, and then updating the module argument.
After you update the module argument, useredoto run the task again with the new args.


Question: Update vars command

Answer:
task_vars[*key*]=*value*updates thetask_vars. You could fix the playbook above by viewing and then updating the task variables instead of the module args.
After you update the task variables, you must useupdate_taskto load the new variables before usingredoto run the task again.


Question: Update task command

Answer: uorupdate_taskrecreates the task from the original task data structure and templates with updated task variables. See the entryUpdate vars commandfor an example of use.


Question: How the debugger interacts with the free strategy

Answer: With the defaultlinearstrategy enabled, Ansible halts execution while the debugger is active, and runs the debugged task immediately after you enter theredocommand. With thefreestrategy enabled, however, Ansible does not wait for all hosts and may queue later tasks on one host before a task fails on another host. With thefreestrategy, Ansible does not queue or execute any tasks while the debugger is active. However, all queued tasks remain in the queue and run as soon as you exit the debugger. If you useredoto reschedule a task from the debugger, other queued tasks may execute before your rescheduled task. For more information about strategies, seeControlling playbook execution: strategies and more.


Question: Asynchronous ad hoc tasks

Answer:
You can execute long-running operations in the background withad hoc tasks. For example, to executelong_running_operationasynchronously in the background, with a timeout (-B) of 3600 seconds, and without polling (-P):
To check on the job status later, use theasync_statusmodule, passing it the job ID that was returned when you ran the original job in the background:
Ansible can also check on the status of your long-running job automatically with polling. In most cases, Ansible will keep the connection to your remote node open between polls. To run for 30 minutes and poll for status every 60 seconds:
Poll mode is smart so all jobs will be started before polling begins on any machine. Be sure to use a high enough--forksvalue if you want to get all of your jobs started very quickly. After the time limit (in seconds) runs out (-B), the process on the remote nodes will be terminated.
Asynchronous mode is best suited to long-running shell commands or software upgrades. Running the copy module asynchronously, for example, does not do a background file transfer.


Question: Asynchronous playbook tasks

Answer: Playbooksalso support asynchronous mode and polling, with a simplified syntax. You can use asynchronous mode in playbooks to avoid connection timeouts or to avoid blocking subsequent tasks. The behavior of asynchronous mode in a playbook depends on the value ofpoll.


Question: Avoid connection timeouts: poll > 0

Answer:
If you want to set a longer timeout limit for a certain task in your playbook, useasyncwithpollset to a positive value. Ansible will still block the next task in your playbook, waiting until the async task either completes, fails or times out. However, the task will only time out if it exceeds the timeout limit you set with theasyncparameter.
To avoid timeouts on a task, specify its maximum runtime and how frequently you would like to poll for status:


Question: Run tasks concurrently: poll = 0

Answer:
If you want to run multiple tasks in a playbook concurrently, useasyncwithpollset to 0. When you setpoll:0, Ansible starts the task and immediately moves on to the next task without waiting for a result. Each async task runs until it either completes, fails or times out (runs longer than itsasyncvalue). The playbook run ends without checking back on async tasks.
To run a playbook task asynchronously:
If you need a synchronization point with an async task, you can register it to obtain its job ID and use theasync_statusmodule to observe it in a later task. For example:
To run multiple asynchronous tasks while limiting the number of tasks running concurrently:


Question: When does it not fit?

Answer:
Prometheus values reliability. You can always view what statistics are
available about your system, even under failure conditions. If you need 100%
accuracy, such as for per-request billing, Prometheus is not a good choice as
the collected data will likely not be detailed and complete enough. In such a
case you would be best off using some other system to collect and analyze the
data for billing, and Prometheus for the rest of your monitoring.


Question: Downloading and running Prometheus

Answer:
Download the latest releaseof Prometheus for
your platform, then extract and run it:
tarxvfzprometheus-*.tar.gzcdprometheus-*
Before starting Prometheus, let's configure it.


Question: Configuring Prometheus to monitor itself

Answer:
Prometheus collects metrics fromtargetsby scraping metrics HTTP
endpoints. Since Prometheus exposes data in the same
manner about itself, it can also scrape and monitor its own health.
While a Prometheus server that collects only data about itself is not very
useful, it is a good starting example. Save the following basic
Prometheus configuration as a file namedprometheus.yml:
global:scrape_interval:15s# By default, scrape targets every 15 seconds.# Attach these labels to any time series or alerts when communicating with# external systems (federation, remote storage, Alertmanager).external_labels:monitor:'codelab-monitor'# A scrape configuration containing exactly one endpoint to scrape:# Here it's Prometheus itself.scrape_configs:# The job name is added as a label `job=
` to any timeseries scraped from this config.-job_name:'prometheus'# Override the global default and scrape targets from this job every 5 seconds.scrape_interval:5sstatic_configs:-targets: ['localhost:9090']
For a complete specification of configuration options, see theconfiguration documentation.


Question: Using the expression browser

Answer:
Let us explore data that Prometheus has collected about itself. To
use Prometheus's built-in expression browser, navigate tohttp://localhost:9090/queryand choose the "Graph" tab.
As you can gather fromlocalhost:9090/metrics,
one metric that Prometheus exports about itself is namedprometheus_target_interval_length_seconds(the actual amount of time between
target scrapes). Enter the below into the expression console and then click "Execute":
prometheus_target_interval_length_seconds
This should return a number of different time series (along with the latest value
recorded for each), each with the metric nameprometheus_target_interval_length_seconds, but with different labels. These
labels designate different latency percentiles and target group intervals.
If we are interested only in 99th percentile latencies, we could use this
query:
prometheus_target_interval_length_seconds{quantile="0.99"}


Question: Using the graphing interface

Answer:
To graph expressions, navigate tohttp://localhost:9090/queryand use the "Graph"
tab.
For example, enter the following expression to graph the per-second rate of chunks
being created in the self-scraped Prometheus:
rate(prometheus_tsdb_head_chunks_created_total[1m])
Experiment with the graph range parameters and other settings.


Question: Starting up some sample targets

Answer:
Let's add additional targets for Prometheus to scrape.
The Node Exporter is used as an example target, for more information on using itsee these instructions.
tar-xzvfnode_exporter-*.*.tar.gzcdnode_exporter-*.*# Start 3 example targets in separate terminals:./node_exporter--web.listen-address127.0.0.1:8080./node_exporter--web.listen-address127.0.0.1:8081./node_exporter--web.listen-address127.0.0.1:8082
You should now have example targets listening onhttp://localhost:8080/metrics,http://localhost:8081/metrics, andhttp://localhost:8082/metrics.


Question: Configure Prometheus to monitor the sample targets

Answer:
Now we will configure Prometheus to scrape these new targets. Let's group all
three endpoints into one job callednode. We will imagine that the
first two endpoints are production targets, while the third one represents a
canary instance. To model this in Prometheus, we can add several groups of
endpoints to a single job, adding extra labels to each group of targets. In
this example, we will add thegroup="production"label to the first group of
targets, while addinggroup="canary"to the second.
To achieve this, add the following job definition to thescrape_configssection in yourprometheus.ymland restart your Prometheus instance:
scrape_configs:-job_name:'node'# Override the global default and scrape targets from this job every 5 seconds.scrape_interval:5sstatic_configs:-targets: ['localhost:8080','localhost:8081']labels:group:'production'-targets: ['localhost:8082']labels:group:'canary'
Go to the expression browser and verify that Prometheus now has information
about time series that these example endpoints expose, such asnode_cpu_seconds_total.


Question: Configure rules for aggregating scraped data into new time series

Answer:
Though not a problem in our example, queries that aggregate over thousands of
time series can get slow when computed ad-hoc. To make this more efficient,
Prometheus can prerecord expressions into new persisted
time series via configuredrecording rules. Let's say we are interested in
recording the per-second rate of cpu time (node_cpu_seconds_total) averaged
over all cpus per instance (but preserving thejob,instanceandmodedimensions) as measured over a window of 5 minutes. We could write this as:
avg by (job, instance, mode) (rate(node_cpu_seconds_total[5m]))
Try graphing this expression.
To record the time series resulting from this expression into a new metric
calledjob_instance_mode:node_cpu_seconds:avg_rate5m, create a file
with the following recording rule and save it asprometheus.rules.yml:
groups:-name:cpu-noderules:-record:job_instance_mode:node_cpu_seconds:avg_rate5mexpr:avg by (job, instance, mode) (rate(node_cpu_seconds_total[5m]))
To make Prometheus pick up this new rule, add arule_filesstatement in yourprometheus.yml. The config should now
look like this:


Question: Reloading configuration

Answer:
As mentioned in theconfiguration documentationa
Prometheus instance can have its configuration reloaded without restarting the
process by using theSIGHUPsignal. If you're running on Linux this can be
performed by usingkill -s SIGHUP
, replacing
with your Prometheus
process ID.


Question: Shutting down your instance gracefully.

Answer:
While Prometheus does have recovery mechanisms in the case that there is an
abrupt process failure it is recommended to use signals or interrupts for a
clean shutdown of a Prometheus instance. On Linux, this can be done by sending
theSIGTERMorSIGINTsignals to the Prometheus process. For example, you
can usekill -s
, replacing
with the signal name
and
with the Prometheus process ID. Alternatively, you can press the
interrupt character at the controlling terminal, which by default is^C(Control-C).


Question: Get started for free

Answer: Sign up and get $200 in credit for your first 60 days with DigitalOcean.*


Question: How to: Python应用领域和职业发展分析

Answer:
简单的说，Python是一个“优雅”、“明确”、“简单”的编程语言。

 - 学习曲线低，非专业人士也能上手
 - 开源系统，拥有强大的生态圈
 - 解释型语言，完美的平台可移植性
 - 动态类型语言，支持面向对象和函数式编程
 - 代码规范程度高，可读性强

Python在以下领域都有用武之地。

 - 后端开发 - Python / Java / Go / PHP
 - DevOps - Python / Shell / Ruby
 - 数据采集 - Python / C++ / Java
 - 量化交易 - Python / C++ / R
 - 数据科学 - Python / R / Julia / Matlab
 - 机器学习 - Python / R / C++ / Julia
 - 自动化测试 - Python / Shell

作为一名Python开发者，根据个人的喜好和职业规划，可以选择的就业领域也非常多。

- Python后端开发工程师（服务器、云平台、数据接口）
- Python运维工程师（自动化运维、SRE、DevOps）
- Python数据分析师（数据分析、商业智能、数字化运营）
- Python数据科学家（机器学习、深度学习、算法专家）
- Python爬虫工程师（不推荐此赛道！！！）
- Python测试工程师（自动化测试、测试开发）

> **说明**：目前，**数据科学赛道是非常热门的方向**，因为不管是互联网行业还是传统行业都已经积累了大量的数据，各行各业都需要数据科学家从已有的数据中发现更多的商业价值，从而为企业的决策提供数据的支撑，这就是所谓的数据驱动决策。

给初学者的几个建议：

- **Make English as your working language.** （让英语成为你的工作语言）
- **Practice makes perfect.** （熟能生巧）
- **All experience comes from the mistakes you've made.** （所有的经验都源于犯过的错误）
- **Don't be a freeloader.** （学会分享，不要只当伸手党）
- （拥抱AI，提升效率）


Question: How to: Day01~20 - Python语言基础

Answer:
#### Day01 - [初识Python](./Day01-20/01.初识Python.md)

1. Python简介
    - Python编年史
    - Python优缺点
    - Python应用领域
2. 安装Python环境
    - Windows环境
    - macOS环境

#### Day02 - [第一个Python程序](./Day01-20/02.第一个Python程序.md)

1. 编写代码的工具
2. 你好世界
3. 注释你的代码

#### Day03 - [Python语言中的变量](./Day01-20/03.Python语言中的变量.md)

1. 一些常识
2. 变量和类型
3. 变量命名
4. 变量的使用

#### Day04 - [Python语言中的运算符](./Day01-20/04.Python语言中的运算符.md)

1. 算术运算符
2. 赋值运算符
3. 比较运算符和逻辑运算符
4. 运算符和表达式应用
    - 华氏和摄氏温度转换
    - 计算圆的周长和面积
    - 判断闰年

#### Day05 - [分支结构](./Day01-20/05.分支结构.md)

1. 使用if和else构造分支结构
2. 使用match和case构造分支结构
3. 分支结构的应用
    - 分段函数求值
    - 百分制成绩转换成等级
    - 计算三角形的周长和面积

#### Day06 - [循环结构](./Day01-20/06.循环结构.md)

1. for-in循环
2. while循环
3. break和continue
4. 嵌套的循环结构
5. 循环结构的应用
    - 判断素数
    - 最大公约数
    - 猜数字游戏

#### Day07 - [分支和循环结构实战](./Day01-20/07.分支和循环结构实战.md)

1. 例子1：100以内的素数
2. 例子2：斐波那契数列
3. 例子3：寻找水仙花数
4. 例子4：百钱百鸡问题
5. 例子5：CRAPS赌博游戏

#### Day08 - [常用数据结构之列表-1](./Day01-20/08.常用数据结构之列表-1.md)

1. 创建列表
2. 列表的运算
3. 元素的遍历

#### Day09 - [常用数据结构之列表-2](./Day01-20/09.常用数据结构之列表-2.md)

1. 列表的方法
    - 添加和删除元素
    - 元素位置和频次
    - 元素排序和反转
2. 列表生成式
3. 嵌套列表
4. 列表的应用

#### Day10 - [常用数据结构之元组](./Day01-20/10.常用数据结构之元组.md)

1. 元组的定义和运算
2. 打包和解包操作
3. 交换变量的值
4. 元组和列表的比较

#### Day11 - [常用数据结构之字符串](./Day01-20/11.常用数据结构之字符串.md)

1. 字符串的定义
    - 转义字符
    - 原始字符串
    - 字符的特殊表示
2. 字符串的运算
    - 拼接和重复
    - 比较运算
    - 成员运算
    - 获取字符串长度
    - 索引和切片
3. 字符的遍历
4. 字符串的方法
    - 大小写相关操作
    - 查找操作
    - 性质判断
    - 格式化
    - 修剪操作
    - 替换操作
    - 拆分与合并
    - 编码与解码
    - 其他方法

#### Day12 - [常用数据结构之集合](./Day01-20/12.常用数据结构之集合.md)

1. 创建集合
2. 元素的变量
3. 集合的运算
    - 成员运算
    - 二元运算
    - 比较运算
4. 集合的方法
5. 不可变集合

#### Day13 - [常用数据结构之字典](./Day01-20/13.常用数据结构之字典.md)

1. 创建和使用字典
2. 字典的运算
3. 字典的方法
4. 字典的应用

#### Day14 - [函数和模块](./Day01-20/14.函数和模块.md)

1. 定义函数
2. 函数的参数
    - 位置参数和关键字参数
    - 参数的默认值
    - 可变参数
3. 用模块管理函数
4. 标准库中的模块和函数

#### Day15 - [函数应用实战](./Day01-20/15.函数应用实战.md)

1. 例子1：随机验证码
2. 例子2：判断素数
3. 例子3：最大公约数和最小公倍数
4. 例子4：数据统计
5. 例子5：双色球随机选号

#### Day16 - [函数使用进阶](./Day01-20/16.函数使用进阶.md)

1. 高阶函数
2. Lambda函数
3. 偏函数

#### Day17 - [函数高级应用](./Day01-20/17.函数高级应用.md)

1. 装饰器
2. 递归调用

#### Day18 - [面向对象编程入门](./Day01-20/18.面向对象编程入门.md)

1. 类和对象
2. 定义类
3. 创建和使用对象
4. 初始化方法
5. 面向对象的支柱
6. 面向对象案例
    - 例子1：数字时钟
    - 例子2：平面上的点

#### Day19 - [面向对象编程进阶](./Day01-20/19.面向对象编程进阶.md)

1. 可见性和属性装饰器
2. 动态属性
3. 静态方法和类方法
4. 继承和多态

#### Day20 - [面向对象编程应用](./Day01-20/20.面向对象编程应用.md)

1. 扑克游戏
2. 工资结算系统


Question: How to: Day21~30 - Python语言应用

Answer:
#### Day21 - [文件读写和异常处理](./Day21-30/21.文件读写和异常处理.md)

1. 打开和关闭文件
2. 读写文本文件
3. 异常处理机制
4. 上下文管理器语法
5. 读写二进制文件

#### Day22 - [对象的序列化和反序列化](./Day21-30/22.对象的序列化和反序列化.md)

1. JSON概述
2. 读写JSON格式的数据
3. 包管理工具pip
4. 使用网络API获取数据

#### Day23 - [Python读写CSV文件](23.Python读写CSV文件.md)

1. CSV文件介绍
2. 将数据写入CSV文件
3. 从CSV文件读取数据

#### Day24 - [Python读写Excel文件-1](./Day21-30/24.用Python读写Excel文件-1.md)

1. Excel简介
2. 读Excel文件
3. 写Excel文件
4. 调整样式
5. 公式计算

#### Day25 - [Python读写Excel文件-2](./Day21-30/25.Python读写Excel文件-2.md)

1. Excel简介
2. 读Excel文件
3. 写Excel文件
4. 调整样式
5. 生成统计图表

#### Day26 - [Python操作Word和PowerPoint文件](./Day21-30/26.Python操作Word和PowerPoint文件.md)

1. 操作Word文档
2. 生成PowerPoint

#### Day27 - [Python操作PDF文件](./Day21-30/27.Python操作PDF文件.md)

1. 从PDF中提取文本
2. 旋转和叠加页面
3. 加密PDF文件
4. 批量添加水印
5. 创建PDF文件

#### Day28 - [Python处理图像](./Day21-30/28.Python处理图像.md)

1. 入门知识
2. 用Pillow处理图像
3. 使用Pillow绘图

#### Day29 - [Python发送邮件和短信](./Day21-30/29.Python发送邮件和短信.md)

1. 发送电子邮件
2. 发送短信

#### Day30 - [正则表达式的应用](./Day21-30/30.正则表达式的应用.md)

1. 正则表达式相关知识
2. Python对正则表达式的支持
    - 例子1：输入验证
    - 例子2：内容提取
    - 例子3：内容替换
    - 例子4：长句拆分


Question: How to: Day31~35 - 其他相关内容

Answer:
#### [Python语言进阶](./Day31-35/31.Python语言进阶.md)

1. 重要知识点
2. 数据结构和算法
3. 函数的使用方式
4. 面向对象相关知识
5. 迭代器和生成器
6. 并发编程

#### [Web前端入门](./Day31-35/32-33.Web前端入门.md)

1. 用HTML标签承载页面内容
2. 用CSS渲染页面
3. 用JavaScript处理交互式行为
5. Vue.js入门
6. Element的使用
7. Bootstrap的使用

#### [玩转Linux操作系统](./Day31-35/34-35.玩转Linux操作系统.md)

1. 操作系统发展史和Linux概述
2. Linux基础命令
3. Linux中的实用程序
4. Linux的文件系统
5. Vim编辑器的应用
6. 环境变量和Shell编程
7. 软件的安装和服务的配置
8. 网络访问和管理
9. 其他相关内容


Question: How to: Day36~45 - 数据库基础和进阶

Answer:
#### Day36 - [关系型数据库和MySQL概述](./Day36-45/36.关系型数据库和MySQL概述.md)

1. 关系型数据库概述
2. MySQL简介
3. 安装MySQL
4. MySQL基本命令

#### Day37 - [SQL详解之DDL](./Day36-45/37.SQL详解之DDL.md)

1. 建库建表
2. 删除表和修改表

#### Day38 - [SQL详解之DML](./Day36-45/38.SQL详解之DML.md)

1. insert操作
2. delete操作
3. update操作

#### Day39 - [SQL详解之DQL](./Day36-45/39.SQL详解之DQL.md)

1. 投影和别名
2. 筛选数据
3. 空值处理
4. 去重
5. 排序
6. 聚合函数
7. 嵌套查询
8. 分组操作
9. 表连接
    - 笛卡尔积
    - 内连接
    - 自然连接
    - 外连接
10. 窗口函数
    - 定义窗口
    - 排名函数
    - 取数函数

#### Day40 - [SQL详解之DCL](./Day36-45/40.SQL详解之DCL.md)

1. 创建用户
2. 授予权限
3. 召回权限

#### Day41 - [MySQL新特性](./Day36-45/41.MySQL新特性.md)

- JSON类型
- 窗口函数
- 公共表表达式

#### Day42 - [视图、函数和过程](./Day36-45/42.视图、函数和过程.md)

1. 视图
    - 使用场景
    - 创建视图
    - 使用限制
2. 函数
    - 内置函数
    - 用户自定义函数（UDF）
3. 过程
    - 创建过程
    - 调用过程

#### Day43 - [索引](./Day36-45/43.索引.md)

1. 执行计划
2. 索引的原理
3. 创建索引
    - 普通索引
    - 唯一索引
    - 前缀索引
    - 复合索引
4. 注意事项

#### Day44 - [Python接入MySQL数据库](./Day36-45/44.Python接入MySQL数据库.md)

1. 安装三方库
2. 创建连接
3. 获取游标
4. 执行SQL语句
5. 通过游标抓取数据
6. 事务提交和回滚
7. 释放连接
8. 编写ETL脚本

#### Day45 - [Hive实战](./Day36-45/45.Hive实战.md)

1. Hive概述
2. 环境搭建
3. 常用命令
4. 基本语法
5. 建表操作
6. 写入数据
7. 常用函数
8. 分组聚合
9. 抽样操作
10. 排序操作
11. 横向展开
12. 性能优化


Question: How to: Day46~60 - 实战Django

Answer:
#### Day46 - [Django快速上手](./Day46-60/46.Django快速上手.md)

1. Web应用工作机制
2. HTTP请求和响应
3. Django框架概述
4. 5分钟快速上手

#### Day47 - [深入模型](./Day46-60/47.深入模型.md)

1. 关系型数据库配置
2. 使用ORM完成对模型的CRUD操作
3. 管理后台的使用
4. Django模型最佳实践
5. 模型定义参考

#### Day48 - [静态资源和Ajax请求](./Day46-60/48.静态资源和Ajax请求.md)

1. 加载静态资源
2. Ajax概述
3. 用Ajax实现投票功能

#### Day49 - [Cookie和Session](./Day46-60/49.Cookie和Session.md)

1. 实现用户跟踪
2. cookie和session的关系
3. Django框架对session的支持
4. 视图函数中的cookie读写操作

#### Day50 - [报表和日志](./Day46-60/50.制作报表.md)

1. 通过`HttpResponse`修改响应头
2. 使用`StreamingHttpResponse`处理大文件
3. 使用`xlwt`生成Excel报表
4. 使用`reportlab`生成PDF报表
5. 使用ECharts生成前端图表

#### Day51 - [日志和调试工具栏](./Day46-60/51.日志和调试工具栏.md)

1. 配置日志
2. 配置Django-Debug-Toolbar
3. 优化ORM代码

#### Day52 - [中间件的应用](./Day46-60/52.中间件的应用.md)

1. 什么是中间件
2. Django框架内置的中间件
3. 自定义中间件及其应用场景

#### Day53 - [前后端分离开发入门](./Day46-60/53.前后端分离开发入门.md)

1. 返回JSON格式的数据
2. 用Vue.js渲染页面

#### Day54 - [RESTful架构和DRF入门](./Day46-60/54.RESTful架构和DRF入门.md)

1. REST概述
2. DRF库使用入门
3. 前后端分离开发
4. JWT的应用

#### Day55 - [RESTful架构和DRF进阶](./Day46-60/55.RESTful架构和DRF进阶.md)

1. 使用CBV
2. 数据分页
3. 数据筛选

#### Day56 - [使用缓存](./Day46-60/56.使用缓存.md)

1. 网站优化第一定律
2. 在Django项目中使用Redis提供缓存服务
3. 在视图函数中读写缓存
4. 使用装饰器实现页面缓存
5. 为数据接口提供缓存服务

#### Day57 - [接入三方平台](./Day46-60/57.接入三方平台.md)

1. 文件上传表单控件和图片文件预览
2. 服务器端如何处理上传的文件

#### Day58 - [异步任务和定时任务](./Day46-60/58.异步任务和定时任务.md)

1. 网站优化第二定律
2. 配置消息队列服务
3. 在项目中使用Celery实现任务异步化
4. 在项目中使用Celery实现定时任务

#### Day59 - [单元测试](./Day46-60/59.单元测试.md)

#### Day60 - [项目上线](./Day46-60/60.项目上线.md)

1. Python中的单元测试
2. Django框架对单元测试的支持
3. 使用版本控制系统
4. 配置和使用uWSGI
5. 动静分离和Nginx配置
6. 配置HTTPS
7. 配置域名解析


Question: How to: Day61~65 - 网络数据采集

Answer:
#### Day61 - [网络数据采集概述](./Day61-65/61.网络数据采集概述.md)

1. 网络爬虫的概念及其应用领域
2. 网络爬虫的合法性探讨
3. 开发网络爬虫的相关工具
4. 一个爬虫程序的构成

#### Day62 - 数据抓取和解析

1. [使用`requests`三方库实现数据抓取](./Day61-65/62.用Python获取网络资源-1.md)
2. [页面解析的三种方式](./Day61-65/62.用Python解析HTML页面-2.md)
    - 正则表达式解析
    - XPath解析
    - CSS选择器解析

#### Day63 - Python中的并发编程

1. [多线程](./Day61-65/63.Python中的并发编程-1.md)
2. [多进程](./Day61-65/63.Python中的并发编程-2.md)
3. [异步I/O](./Day61-65/63.Python中的并发编程-3.md)

#### Day64 - [使用Selenium抓取网页动态内容](./Day61-65/64.使用Selenium抓取网页动态内容.md)

1. 安装Selenium
2. 加载页面
3. 查找元素和模拟用户行为
4. 隐式等待和显示等待
5. 执行JavaScript代码
6. Selenium反爬破解
7. 设置无头浏览器

#### Day65 - [爬虫框架Scrapy简介](./Day61-65/65.爬虫框架Scrapy简介.md)

1. Scrapy核心组件
2. Scrapy工作流程
3. 安装Scrapy和创建项目
4. 编写蜘蛛程序
5. 编写中间件和管道程序
6. Scrapy配置文件


Question: How to: Day66~80 - Python数据分析

Answer:
#### Day66 - [数据分析概述](./Day66-80/66.数据分析概述.md)

1. 数据分析师的职责
2. 数据分析师的技能栈
3. 数据分析相关库

#### Day67 - [环境准备](./Day66-80/67.环境准备.md)

1. 安装和使用anaconda
    - conda相关命令
2. 安装和使用jupyter-lab
    - 安装和启动
    - 使用小技巧

#### Day68 - [NumPy的应用-1](./Day66-80/68.NumPy的应用-1.md)

1. 创建数组对象
2. 数组对象的属性
3. 数组对象的索引运算
    - 普通索引
    - 花式索引
    - 布尔索引
    - 切片索引
4. 案例：使用数组处理图像

#### Day69 - [NumPy的应用-2](./Day66-80/69.NumPy的应用-2.md)

1. 数组对象的相关方法
    - 获取描述性统计信息
    - 其他相关方法

#### Day70 - [NumPy的应用-3](./Day66-80/70.NumPy的应用-3.md)

1. 数组的运算
    - 数组跟标量的运算
    - 数组跟数组的运算
2. 通用一元函数
3. 通用二元函数
4. 广播机制
5. Numpy常用函数

#### Day71 - [NumPy的应用-4](./Day66-80/71.NumPy的应用-4.md)

1. 向量
2. 行列式
3. 矩阵
4. 多项式

#### Day72 - [深入浅出pandas-1](./Day66-80/72.深入浅出pandas-1.md)

1. 创建Series对象
2. Series对象的运算
3. Series对象的属性和方法

#### Day73 - [深入浅出pandas-2](./Day66-80/73.深入浅出pandas-2.md)

1. 创建DataFrame对象
2. DataFrame对象的属性和方法
3. 读写DataFrame中的数据

#### Day74 - [深入浅出pandas-3](./Day66-80/74.深入浅出pandas-3.md)

1. 数据重塑
    - 数据拼接
    - 数据合并
2. 数据清洗
    - 缺失值
    - 重复值
    - 异常值
    - 预处理

#### Day75 - [深入浅出pandas-4](./Day66-80/75.深入浅出pandas-4.md)

1. 数据透视
    - 获取描述性统计信息
    - 排序和头部值
    - 分组聚合
    - 透视表和交叉表
2. 数据呈现

#### Day76 - [深入浅出pandas-5](./Day66-80/76.深入浅出pandas-5.md)

1. 计算同比环比
2. 窗口计算
3. 相关性判定

#### Day77 - [深入浅出pandas-6](./Day66-80/77.深入浅出pandas-6.md)

1. 索引的使用
    - 范围索引
    - 分类索引
    - 多级索引
    - 间隔索引
    - 日期时间索引

#### Day78 - [数据可视化-1](./Day66-80/78.数据可视化-1.md)

1. 安装和导入matplotlib
2. 创建画布
3. 创建坐标系
4. 绘制图表
    - 折线图
    - 散点图
    - 柱状图
    - 饼状图
    - 直方图
    - 箱线图
5. 显示和保存图表

#### Day79 - [数据可视化-2](./Day66-80/79.数据可视化-2.md)

1. 高阶图表
    - 气泡图
    - 面积图
    - 雷达图
    - 玫瑰图
    - 3D图表

#### Day80 - [数据可视化-3](./Day66-80/80.数据可视化-3.md)

1. Seaborn
2. Pyecharts


Question: How to: Day81~90 - 机器学习

Answer:
#### Day81 - [浅谈机器学习](./Day81-90/81.浅谈机器学习.md)

1. 人工智能发展史
2. 什么是机器学习
3. 机器学习应用领域
4. 机器学习的分类
5. 机器学习的步骤
6. 第一次机器学习

#### Day82 - [k最近邻算法](./Day81-90/82.k最近邻算法.md)

1. 距离的度量
2. 数据集介绍
3. kNN分类的实现
4. 模型评估
5. 参数调优
6. kNN回归的实现

#### Day83 - [决策树和随机森林](./Day81-90/83.决策树和随机森林.md)

1. 决策树的构建
    - 特征选择
    - 数据分裂
    - 树的剪枝
2. 实现决策树模型
3. 随机森林概述

#### Day84 - [朴素贝叶斯算法](./Day81-90/84.朴素贝叶斯算法.md)

1. 贝叶斯定理
2. 朴素贝叶斯
3. 算法原理
    - 训练阶段
    - 预测阶段
    - 代码实现
4. 算法优缺点

#### Day85 - [回归模型](./Day81-90/85.回归模型.md)

1. 回归模型的分类
2. 回归系数的计算
3. 新数据集介绍
4. 线性回归代码实现
5. 回归模型的评估
6. 引入正则化项
7. 线性回归另一种实现
8. 多项式回归
9. 逻辑回归

#### Day86 - [K-Means聚类算法](./Day81-90/86.K-Means聚类算法.md)

1. 算法原理
2. 数学描述
3. 代码实现

#### Day87 - [集成学习算法](./Day81-90/87.集成学习算法.md)

1. 算法分类
2. AdaBoost
3. GBDT
4. XGBoost
5. LightGBM

#### Day88 - [神经网络模型](./Day81-90/88.神经网络模型.md)

1. 基本构成
2. 工作原理
3. 代码实现
4. 模型优缺点

#### Day89 - [自然语言处理入门](./Day81-90/89.自然语言处理入门.md)

1. 词袋模型
2. 词向量
3. NPLM和RNN
4. Seq2Seq
5. Transformer

#### Day90 - [机器学习实战](./Day81-90/90.机器学习实战.md)

1. 数据探索
1. 特征工程
1. 模型训练
1. 模型评估
1. 模型部署


Question: How to: Day91~99 - [团队项目开发](./Day91-100)

Answer:
#### 第91天：[团队项目开发的问题和解决方案](./Day91-100/91.团队项目开发的问题和解决方案.md)

1. 软件过程模型
   - 经典过程模型（瀑布模型）
     - 可行性分析（研究做还是不做），输出《可行性分析报告》。
     - 需求分析（研究做什么），输出《需求规格说明书》和产品界面原型图。
     - 概要设计和详细设计，输出概念模型图（ER图）、物理模型图、类图、时序图等。
     - 编码 / 测试。
     - 上线 / 维护。

     瀑布模型最大的缺点是无法拥抱需求变化，整套流程结束后才能看到产品，团队士气低落。
   - 敏捷开发（Scrum）- 产品所有者、Scrum Master、研发人员 - Sprint
     - 产品的Backlog（用户故事、产品原型）。
     - 计划会议（评估和预算）。
     - 日常开发（站立会议、番茄工作法、结对编程、测试先行、代码重构……）。
     - 修复bug（问题描述、重现步骤、测试人员、被指派人）。
     - 发布版本。
     - 评审会议（Showcase，用户需要参与）。
     - 回顾会议（对当前迭代周期做一个总结）。

     > 补充：敏捷软件开发宣言
     >
     > - **个体和互动** 高于 流程和工具
     > - **工作的软件** 高于 详尽的文档
     > - **客户合作** 高于 合同谈判
     > - **响应变化** 高于 遵循计划

     ![](./res/agile-scrum-sprint-cycle.png)

     > 角色：产品所有者（决定做什么，能对需求拍板的人）、团队负责人（解决各种问题，专注如何更好的工作，屏蔽外部对开发团队的影响）、开发团队（项目执行人员，具体指开发人员和测试人员）。

     > 准备工作：商业案例和资金、合同、憧憬、初始产品需求、初始发布计划、入股、组建团队。

     >  敏捷团队通常人数为8-10人。

     >  工作量估算：将开发任务量化，包括原型、Logo设计、UI设计、前端开发等，尽量把每个工作分解到最小任务量，最小任务量标准为工作时间不能超过两天，然后估算总体项目时间。把每个任务都贴在看板上面，看板上分三部分：to do（待完成）、in progress（进行中）和done（已完成）。

2. 项目团队组建

   - 团队的构成和角色

     ![company_architecture](./res/company_architecture.png)

   - 编程规范和代码审查（`flake8`、`pylint`）

     ![](./res/pylint.png)

   - Python中的一些“惯例”（请参考[《Python惯例-如何编写Pythonic的代码》](./番外篇/Python编程惯例.md)）

   - 影响代码可读性的原因：

     - 代码注释太少或者没有注释
     - 代码破坏了语言的最佳实践
     - 反模式编程（意大利面代码、复制-黏贴编程、自负编程、……）

3. 团队开发工具介绍
   - 版本控制：Git、Mercury
   - 缺陷管理：[Gitlab](https://about.gitlab.com/)、[Redmine](http://www.redmine.org.cn/)
   - 敏捷闭环工具：[禅道](https://www.zentao.net/)、[JIRA](https://www.atlassian.com/software/jira/features)
   - 持续集成：[Jenkins](https://jenkins.io/)、[Travis-CI](https://travis-ci.org/)

   请参考[《团队项目开发的问题和解决方案》](Day91-100/91.团队项目开发的问题和解决方案.md)。

##### 项目选题和理解业务

1. 选题范围设定

   - CMS（用户端）：新闻聚合网站、问答/分享社区、影评/书评网站等。
   - MIS（用户端+管理端）：KMS、KPI考核系统、HRS、CRM系统、供应链系统、仓储管理系统等。

   - App后台（管理端+数据接口）：二手交易类、报刊杂志类、小众电商类、新闻资讯类、旅游类、社交类、阅读类等。
   - 其他类型：自身行业背景和工作经验、业务容易理解和把控。

2. 需求理解、模块划分和任务分配

   - 需求理解：头脑风暴和竞品分析。
   - 模块划分：画思维导图（XMind），每个模块是一个枝节点，每个具体的功能是一个叶节点（用动词表述），需要确保每个叶节点无法再生出新节点，确定每个叶子节点的重要性、优先级和工作量。
   - 任务分配：由项目负责人根据上面的指标为每个团队成员分配任务。

   ![](./res/requirements_by_xmind.png)

3. 制定项目进度表（每日更新）

   | 模块 | 功能     | 人员   | 状态     | 完成 | 工时 | 计划开始 | 实际开始 | 计划结束 | 实际结束 | 备注             |
   | ---- | -------- | ------ | -------- | ---- | ---- | -------- | -------- | -------- | -------- | ---------------- |
   | 评论 | 添加评论 | 王大锤 | 正在进行 | 50%  | 4    | 2018/8/7 |          | 2018/8/7 |          |                  |
   |      | 删除评论 | 王大锤 | 等待     | 0%   | 2    | 2018/8/7 |          | 2018/8/7 |          |                  |
   |      | 查看评论 | 白元芳 | 正在进行 | 20%  | 4    | 2018/8/7 |          | 2018/8/7 |          | 需要进行代码审查 |
   |      | 评论投票 | 白元芳 | 等待     | 0%   | 4    | 2018/8/8 |          | 2018/8/8 |          |                  |

4. OOAD和数据库设计

  - UML（统一建模语言）的类图

    ![uml](./res/uml-class-diagram.png)

  - 通过模型创建表（正向工程），例如在Django项目中可以通过下面的命令创建二维表。

    ```Shell
    python manage.py makemigrations app
    python manage.py migrate
    ```

  - 使用PowerDesigner绘制物理模型图。

    ![](./res/power-designer-pdm.png)

  - 通过数据表创建模型（反向工程），例如在Django项目中可以通过下面的命令生成模型。

    ```Shell
    python manage.py inspectdb > app/models.py
    ```

#### 第92天：[Docker容器技术详解](./Day91-100/92.Docker容器技术详解.md)

1. Docker简介
2. 安装Docker
3. 使用Docker创建容器（Nginx、MySQL、Redis、Gitlab、Jenkins）
4. 构建Docker镜像（Dockerfile的编写和相关指令）
5. 容器编排（Docker-compose）
6. 集群管理（Kubernetes）

#### 第93天：[MySQL性能优化](./Day91-100/93.MySQL性能优化.md)

1. 基本原则
2. InnoDB引擎
3. 索引的使用和注意事项
4. 数据分区
5. SQL优化
6. 配置优化
7. 架构优化

#### 第94天：[网络API接口设计](./Day91-100/94.网络API接口设计.md)

1. 设计原则
    - 关键问题
    - 其他问题
2. 文档撰写

#### 第95天：[使用Django开发商业项目](./Day91-100/95.使用Django开发商业项	目.md)

##### 项目开发中的公共问题

1. 数据库的配置（多数据库、主从复制、数据库路由）
2. 缓存的配置（分区缓存、键设置、超时设置、主从复制、故障恢复（哨兵））
3. 日志的配置
4. 分析和调试（Django-Debug-ToolBar）
5. 好用的Python模块（日期计算、图像处理、数据加密、三方API）

##### REST API设计

1. RESTful架构
   - [理解RESTful架构](http://www.ruanyifeng.com/blog/2011/09/restful.html)
   - [RESTful API设计指南](http://www.ruanyifeng.com/blog/2014/05/restful_api.html)
   - [RESTful API最佳实践](http://www.ruanyifeng.com/blog/2018/10/restful-api-best-practices.html)
2. API接口文档的撰写
   - [RAP2](http://rap2.taobao.org/)
   - [YAPI](http://yapi.demo.qunar.com/)
3. [django-REST-framework](https://www.django-rest-framework.org/)的应用

##### 项目中的重点难点剖析

1. 使用缓存缓解数据库压力 - Redis
2. 使用消息队列做解耦合和削峰 - Celery + RabbitMQ

#### 第96天：[软件测试和自动化测试](Day91-100/96.软件测试和自动化测试.md)

##### 单元测试

1. 测试的种类
2. 编写单元测试（`unittest`、`pytest`、`nose2`、`tox`、`ddt`、……）
3. 测试覆盖率（`coverage`）

##### Django项目部署

1. 部署前的准备工作
   - 关键设置（SECRET_KEY / DEBUG / ALLOWED_HOSTS / 缓存 / 数据库）
   - HTTPS / CSRF_COOKIE_SECUR  / SESSION_COOKIE_SECURE
   - 日志相关配置
2. Linux常用命令回顾
3. Linux常用服务的安装和配置
4. uWSGI/Gunicorn和Nginx的使用
   - Gunicorn和uWSGI的比较
     - 对于不需要大量定制化的简单应用程序，Gunicorn是一个不错的选择，uWSGI的学习曲线比Gunicorn要陡峭得多，Gunicorn的默认参数就已经能够适应大多数应用程序。
     - uWSGI支持异构部署。
     - 由于Nginx本身支持uWSGI，在线上一般都将Nginx和uWSGI捆绑在一起部署，而且uWSGI属于功能齐全且高度定制的WSGI中间件。
     - 在性能上，Gunicorn和uWSGI其实表现相当。
5. 使用虚拟


Question: How to: 第100天 - [补充内容](./Day91-100/100.补充内容.md)

Answer:
- 面试宝典
    - Python 面试宝典
    - SQL 面试宝典（数据分析师）
    - 商业分析面试宝典
    - 机器学习面试宝典

- 机器学习数学基础
- 深度学习
    - 计算机视觉
    - 大语言模型


Question: How to: Scalability

Answer:
* [Microservices and Orchestration](https://martinfowler.com/microservices/)
	* [Domain-Oriented Microservice Architecture at Uber](https://eng.uber.com/microservice-architecture/)
	* [Service Architecture (3 parts: Domain Gateways, Value-Added Services, BFF) at SoundCloud](https://developers.soundcloud.com/blog/service-architecture-3)
	* [Container (8 parts) at Riot Games](https://engineering.riotgames.com/news/thinking-inside-container)
	* [Containerization at Pinterest](https://medium.com/@Pinterest_Engineering/containerization-at-pinterest-92295347f2f3)
	* [Evolution of Container Usage at Netflix](https://medium.com/netflix-techblog/the-evolution-of-container-usage-at-netflix-3abfc096781b)
	* [Dockerizing MySQL at Uber](https://eng.uber.com/dockerizing-mysql/)
	* [Testing of Microservices at Spotify](https://labs.spotify.com/2018/01/11/testing-of-microservices/)
	* [Docker in Production at Treehouse](https://medium.com/treehouse-engineering/lessons-learned-running-docker-in-production-5dce99ece770)
	* [Microservice at SoundCloud](https://developers.soundcloud.com/blog/inside-a-soundcloud-microservice)
	* [Operate Kubernetes Reliably at Stripe](https://stripe.com/blog/operating-kubernetes)
	* [Cross-Cluster Traffic Mirroring with Istio at Trivago](https://tech.trivago.com/2020/06/10/cross-cluster-traffic-mirroring-with-istio/)
	* [Agrarian-Scale Kubernetes (3 parts) at New York Times](https://open.nytimes.com/agrarian-scale-kubernetes-part-3-ee459887ed7e)
	* [Nanoservices at BBC](https://medium.com/bbc-design-engineering/powering-bbc-online-with-nanoservices-727840ba015b)
	* [PowerfulSeal: Testing Tool for Kubernetes Clusters at Bloomberg](https://www.techatbloomberg.com/blog/powerfulseal-testing-tool-kubernetes-clusters/)
	* [Conductor: Microservices Orchestrator at Netflix](https://medium.com/netflix-techblog/netflix-conductor-a-microservices-orchestrator-2e8d4771bf40)
	* [Docker Containers that Power Over 100.000 Online Shops at Shopify](https://shopifyengineering.myshopify.com/blogs/engineering/docker-at-shopify-how-we-built-containers-that-power-over-100-000-online-shops)
	* [Microservice Architecture at Medium](https://medium.engineering/microservice-architecture-at-medium-9c33805eb74f)
	* [From bare-metal to Kubernetes at Betabrand](https://boxunix.com/post/bare_metal_to_kube/)
	* [Kubernetes at Tinder](https://medium.com/tinder-engineering/tinders-move-to-kubernetes-cda2a6372f44)
	* [Kubernetes at Quora](https://www.quora.com/q/quoraengineering/Adopting-Kubernetes-at-Quora)
	* [Kubernetes Platform at Pinterest](https://medium.com/pinterest-engineering/building-a-kubernetes-platform-at-pinterest-fb3d9571c948)
	* [Microservices at Nubank](https://medium.com/building-nubank/microservices-at-nubank-an-overview-2ebcb336c64d)
	* [Payment Transaction Management in Microservices at Mercari](https://engineering.mercari.com/en/blog/entry/20210831-2019-06-07-155849/)
	* [Service Mesh at Snap](https://eng.snap.com/monolith-to-multicloud-microservices-snap-service-mesh)
	* [GRIT: Protocol for Distributed Transactions across Microservices at eBay](https://tech.ebayinc.com/engineering/grit-a-protocol-for-distributed-transactions-across-microservices/)
	* [Rubix: Kubernetes at Palantir](https://medium.com/palantir/introducing-rubix-kubernetes-at-palantir-ab0ce16ea42e)
	* [CRISP: Critical Path Analysis for Microservice Architectures at Uber](https://eng.uber.com/crisp-critical-path-analysis-for-microservice-architectures/)
* [Distributed Caching](https://www.wix.engineering/post/scaling-to-100m-to-cache-or-not-to-cache)
	* [EVCache: Distributed In-memory Caching at Netflix](https://medium.com/netflix-techblog/caching-for-a-global-netflix-7bcc457012f1)
	* [EVCache Cache Warmer Infrastructure at Netflix](https://medium.com/netflix-techblog/cache-warming-agility-for-a-stateful-service-2d3b1da82642)
	* [Memsniff: Robust Memcache Traffic Analyzer at Box](https://blog.box.com/blog/introducing-memsniff-robust-memcache-traffic-analyzer/)
	* [Caching with Consistent Hashing and Cache Smearing at Etsy](https://codeascraft.com/2017/11/30/how-etsy-caches/)
	* [Analysis of Photo Caching at Facebook](https://code.facebook.com/posts/220956754772273/an-analysis-of-facebook-photo-caching/)
	* [Cache Efficiency Exercise at Facebook](https://code.facebook.com/posts/964122680272229/web-performance-cache-efficiency-exercise/)
	* [tCache: Scalable Data-aware Java Caching at Trivago](http://tech.trivago.com/2015/10/15/tcache/)
	* [Pycache: In-process Caching at Quora](https://quoraengineering.quora.com/Pycache-lightning-fast-in-process-caching)
	* [Reduce Memcached Memory Usage by 50% at Trivago](http://tech.trivago.com/2017/12/19/how-trivago-reduced-memcached-memory-usage-by-50/)
	* [Caching Internal Service Calls at Yelp](https://engineeringblog.yelp.com/2018/03/caching-internal-service-calls-at-yelp.html)
	* [Estimating the Cache Efficiency using Big Data at Allegro](https://allegro.tech/2017/01/estimating-the-cache-efficiency-usin


Question: How to: Availability

Answer:
* [Resilience Engineering: Learning to Embrace Failure](https://queue.acm.org/detail.cfm?id=2371297)
	* [Resilience Engineering with Project Waterbear at LinkedIn](https://engineering.linkedin.com/blog/2017/11/resilience-engineering-at-linkedin-with-project-waterbear)
	* [Resiliency against Traffic Oversaturation at iHeartRadio](https://tech.iheart.com/resiliency-against-traffic-oversaturation-77c5ed92a5fb)
	* [Resiliency in Distributed Systems at GO-JEK](https://blog.gojekengineering.com/resiliency-in-distributed-systems-efd30f74baf4)
	* [Practical NoSQL Resilience Design Pattern for the Enterprise at eBay](https://www.ebayinc.com/stories/blogs/tech/practical-nosql-resilience-design-pattern-for-the-enterprise/)
	* [Ensuring Resilience to Disaster at Quora](https://quoraengineering.quora.com/Ensuring-Quoras-Resilience-to-Disaster)
	* [Site Resiliency at Expedia](https://www.infoq.com/presentations/expedia-website-resiliency?utm_source=presentations_about_Case_Study&utm_medium=link&utm_campaign=Case_Study)
	* [Resiliency and Disaster Recovery with Kafka at eBay](https://tech.ebayinc.com/engineering/resiliency-and-disaster-recovery-with-kafka/)
	* [Disaster Recovery for Multi-Region Kafka at Uber](https://eng.uber.com/kafka/)
* [Failover](http://cloudpatterns.org/mechanisms/failover_system)
	* [The Evolution of Global Traffic Routing and Failover](https://www.usenix.org/conference/srecon16/program/presentation/heady)
	* [Testing for Disaster Recovery Failover Testing](https://www.usenix.org/conference/srecon17asia/program/presentation/liu_zehua)
	* [Designing a Microservices Architecture for Failure](https://blog.risingstack.com/designing-microservices-architecture-for-failure/)
	* [ELB for Automatic Failover at GoSquared](https://engineering.gosquared.com/use-elb-automatic-failover)
	* [Eliminate the Database for Higher Availability at American Express](http://americanexpress.io/eliminate-the-database-for-higher-availability/)
	* [Failover with Redis Sentinel at Vinted](http://engineering.vinted.com/2015/09/03/failover-with-redis-sentinel/)
	* [High-availability SaaS Infrastructure at FreeAgent](http://engineering.freeagent.com/2017/02/06/ha-infrastructure-without-breaking-the-bank/)
	* [MySQL High Availability at GitHub](https://github.blog/2018-06-20-mysql-high-availability-at-github/)
	* [MySQL High Availability at Eventbrite](https://www.eventbrite.com/engineering/mysql-high-availability-at-eventbrite/)
	* [Business Continuity & Disaster Recovery at Walmart](https://medium.com/walmartlabs/business-continuity-disaster-recovery-in-the-microservices-world-ef2adca363df)
* [Load Balancing](https://blog.vivekpanyam.com/scaling-a-web-service-load-balancing/)
	* [Introduction to Modern Network Load Balancing and Proxying](https://blog.envoyproxy.io/introduction-to-modern-network-load-balancing-and-proxying-a57f6ff80236)
	* [Top Five (Load Balancing) Scalability Patterns](https://www.f5.com/company/blog/top-five-scalability-patterns)
	* [Load Balancing infrastructure to support more than 1.3 billion users at Facebook](https://www.usenix.org/conference/srecon15europe/program/presentation/shuff)
	* [DHCPLB: DHCP Load Balancer at Facebook](https://code.facebook.com/posts/1734309626831603/dhcplb-an-open-source-load-balancer/)
	* [Katran: Scalable Network Load Balancer at Facebook](https://code.facebook.com/posts/1906146702752923/open-sourcing-katran-a-scalable-network-load-balancer/)
	* [Deterministic Aperture: A Distributed, Load Balancing Algorithm at Twitter](https://blog.twitter.com/engineering/en_us/topics/infrastructure/2019/daperture-load-balancer.html)
	* [Load Balancing with Eureka at Netflix](https://medium.com/netflix-techblog/netflix-shares-cloud-load-balancing-and-failover-tool-eureka-c10647ef95e5)
	* [Edge Load Balancing at Netflix](https://medium.com/netflix-techblog/netflix-edge-load-balancing-695308b5548c)
	* [Zuul 2: Cloud Gateway at Netflix](https://medium.com/netflix-techblog/open-sourcing-zuul-2-82ea476cb2b3)
	* [Load Balancing at Yelp](https://engineeringblog.yelp.com/2017/05/taking-zero-downtime-load-balancing-even-further.html)
	* [Load Balancing at Github](https://githubengineering.com/introducing-glb/)
	* [Consistent Hashing to Improve Load Balancing at Vimeo](https://medium.com/vimeo-engineering-blog/improving-load-balancing-with-a-new-consistent-hashing-algorithm-9f1bd75709ed)
	* [UDP Load Balancing at 500 pixel](https://developers.500px.com/udp-load-balancing-with-keepalived-167382d7ad08)
	* [QALM: QoS Load Management Framework at Uber](https://eng.uber.com/qalm/)
	* [Traffic Steering using Rum DNS at LinkedIn](https://www.usenix.org/conference/srecon17europe/program/presentation/rastogi)
	* [Traffic Infrastructure (Edge Network) at Dropbox](https://dropbox.tech/infrastructure/dropbox-traffic-infrastructure-edge-network)
	* [Intelligent DNS based load balancing at Dropbox](https://dropbox.tech/infrastructure/intelligent-dns-based-load-balancing-at-dropbox)
	* [Monitor DNS systems at St


Question: How to: Performance

Answer:
* [Performance Optimization on OS, Storage, Database, Network](https://stackify.com/application-performance-metrics/)
	* [Improving Performance with Background Data Prefetching at Instagram](https://engineering.instagram.com/improving-performance-with-background-data-prefetching-b191acb39898)
	* [Fixing Linux filesystem performance regressions at LinkedIn](https://engineering.linkedin.com/blog/2020/fixing-linux-filesystem-performance-regressions)
	* [Compression Techniques to Solve Network I/O Bottlenecks at eBay](https://www.ebayinc.com/stories/blogs/tech/how-ebays-shopping-cart-used-compression-techniques-to-solve-network-io-bottlenecks/)
	* [Optimizing Web Servers for High Throughput and Low Latency at Dropbox](https://dropbox.tech/infrastructure/optimizing-web-servers-for-high-throughput-and-low-latency)
	* [Linux Performance Analysis in 60.000 Milliseconds at Netflix](https://medium.com/netflix-techblog/linux-performance-analysis-in-60-000-milliseconds-accc10403c55)
	* [Live Downsizing Google Cloud Persistent Disks (PD-SSD) at Mixpanel](https://engineering.mixpanel.com/2018/07/31/live-downsizing-google-cloud-pds-for-fun-and-profit/)
	* [Decreasing RAM Usage by 40% Using jemalloc with Python & Celery at Zapier](https://zapier.com/engineering/celery-python-jemalloc/)
	* [Reducing Memory Footprint at Slack](https://slack.engineering/reducing-slacks-memory-footprint-4480fec7e8eb)
	* [Continuous Load Testing at Slack](https://slack.engineering/continuous-load-testing/)
	* [Performance Improvements at Pinterest](https://medium.com/@Pinterest_Engineering/driving-user-growth-with-performance-improvements-cfc50dafadd7)
	* [Server Side Rendering at Wix](https://www.youtube.com/watch?v=f9xI2jR71Ms)
	* [30x Performance Improvements on MySQLStreamer at Yelp](https://engineeringblog.yelp.com/2018/02/making-30x-performance-improvements-on-yelps-mysqlstreamer.html)
	* [Optimizing APIs at Netflix](https://medium.com/netflix-techblog/optimizing-the-netflix-api-5c9ac715cf19)
	* [Performance Monitoring with Riemann and Clojure at Walmart](https://medium.com/walmartlabs/performance-monitoring-with-riemann-and-clojure-eafc07fcd375)
	* [Performance Tracking Dashboard for Live Games at Zynga](https://www.zynga.com/blogs/engineering/live-games-have-evolving-performance)
	* [Optimizing CAL Report Hadoop MapReduce Jobs at eBay](https://www.ebayinc.com/stories/blogs/tech/optimization-of-cal-report-hadoop-mapreduce-job/)
	* [Performance Tuning on Quartz Scheduler at eBay](https://www.ebayinc.com/stories/blogs/tech/performance-tuning-on-quartz-scheduler/)
	* [Profiling C++ (Part 1: Optimization, Part 2: Measurement and Analysis) at Riot Games](https://engineering.riotgames.com/news/profiling-optimisation)
	* [Profiling React Server-Side Rendering at HomeAway](https://medium.com/homeaway-tech-blog/profiling-react-server-side-rendering-to-free-the-node-js-event-loop-7f0fe455a901)
	* [Hardware-Assisted Video Transcoding at Dailymotion](https://medium.com/dailymotion-engineering/hardware-assisted-video-transcoding-at-dailymotion-66cd2db448ae)
	* [Cross Shard Transactions at 10 Million RPS at Dropbox](https://dropbox.tech/infrastructure/cross-shard-transactions-at-10-million-requests-per-second)
	* [API Profiling at Pinterest](https://medium.com/@Pinterest_Engineering/api-profiling-at-pinterest-6fa9333b4961)
	* [Pagelets Parallelize Server-side Processing at Yelp](https://engineeringblog.yelp.com/2017/07/generating-web-pages-in-parallel-with-pagelets.html)
	* [Improving key expiration in Redis at Twitter](https://blog.twitter.com/engineering/en_us/topics/infrastructure/2019/improving-key-expiration-in-redis.html)
	* [Ad Delivery Network Performance Optimization with Flame Graphs at MindGeek](https://medium.com/mindgeek-engineering-blog/ad-delivery-network-performance-optimization-with-flame-graphs-bc550cf59cf7)
	* [Predictive CPU isolation of containers at Netflix](https://medium.com/netflix-techblog/predictive-cpu-isolation-of-containers-at-netflix-91f014d856c7)
	* [Improving HDFS I/O Utilization for Efficiency at Uber](https://eng.uber.com/improving-hdfs-i-o-utilization-for-efficiency/)
	* [Cloud Jewels: Estimating kWh in the Cloud at Etsy](https://codeascraft.com/2020/04/23/cloud-jewels-estimating-kwh-in-the-cloud/)
	* [Unthrottled: Fixing CPU Limits in the Cloud (2 parts) at Indeed](https://engineering.indeedblog.com/blog/2019/12/unthrottled-fixing-cpu-limits-in-the-cloud/)
* [Performance Optimization by Tuning Garbage Collection](https://confluence.atlassian.com/enterprise/garbage-collection-gc-tuning-guide-461504616.html)
	* [Garbage Collection in Java Applications at LinkedIn](https://engineering.linkedin.com/garbage-collection/garbage-collection-optimization-high-throughput-and-low-latency-java-applications)
	* [Garbage Collection in High-Throughput, Low-Latency Machine Learning Services at Adobe](https://medium.com/adobetech/engineering-high-throughput-low-latency-machine-learning-services-7d45edac0271)
	* [Garbage Collection i


Question: How to: Intelligence

Answer:
* [Big Data](https://insights.sei.cmu.edu/sei_blog/2017/05/reference-architectures-for-big-data-systems.html)
	* [Data Platform at Uber](https://eng.uber.com/uber-big-data-platform/)
	* [Data Platform at BMW](https://www.unibw.de/code/events-u/jt-2018-workshops/ws3_bigdata_vortrag_widmann.pdf)
	* [Data Platform at Netflix](https://www.youtube.com/watch?v=CSDIThSwA7s)
	* [Data Platform at Flipkart](https://blog.flipkart.tech/overview-of-flipkart-data-platform-20c6d3e9a196)
	* [Data Platform at Coupang](https://medium.com/coupang-tech/evolving-the-coupang-data-platform-308e305a9c45)
	* [Data Platform at DoorDash](https://doordash.engineering/2020/09/25/how-doordash-is-scaling-its-data-platform/)
	* [Data Platform at Khan Academy](http://engineering.khanacademy.org/posts/khanalytics.htm)
	* [Data Infrastructure at Airbnb](https://medium.com/airbnb-engineering/data-infrastructure-at-airbnb-8adfb34f169c)
	* [Data Infrastructure at LinkedIn](https://www.infoq.com/presentations/big-data-infrastructure-linkedin)
	* [Data Infrastructure at GO-JEK](https://blog.gojekengineering.com/data-infrastructure-at-go-jek-cd4dc8cbd929)
	* [Data Ingestion Infrastructure at Pinterest](https://medium.com/@Pinterest_Engineering/scalable-and-reliable-data-ingestion-at-pinterest-b921c2ee8754)
	* [Data Analytics Architecture at Pinterest](https://medium.com/@Pinterest_Engineering/behind-the-pins-building-analytics-f7b508cdacab)
	* [Data Orchestration Service at Spotify](https://engineering.atspotify.com/2022/03/why-we-switched-our-data-orchestration-service/)
	* [Big Data Processing (2 parts) at Spotify](https://labs.spotify.com/2017/10/23/big-data-processing-at-spotify-the-road-to-scio-part-2/)
	* [Big Data Processing at Uber](https://cdn.oreillystatic.com/en/assets/1/event/160/Big%20data%20processing%20with%20Hadoop%20and%20Spark%2C%20the%20Uber%20way%20Presentation.pdf)
	* [Analytics Pipeline at Lyft](https://cdn.oreillystatic.com/en/assets/1/event/269/Lyft_s%20analytics%20pipeline_%20From%20Redshift%20to%20Apache%20Hive%20and%20Presto%20Presentation.pdf)
	* [Analytics Pipeline at Grammarly](https://tech.grammarly.com/blog/building-a-versatile-analytics-pipeline-on-top-of-apache-spark)
	* [Analytics Pipeline at Teads](https://medium.com/teads-engineering/give-meaning-to-100-billion-analytics-events-a-day-d6ba09aa8f44)
	* [ML Data Pipelines for Real-Time Fraud Prevention at PayPal](https://www.infoq.com/presentations/paypal-ml-fraud-prevention-2018)
	* [Big Data Analytics and ML Techniques at LinkedIn](https://cdn.oreillystatic.com/en/assets/1/event/269/Big%20data%20analytics%20and%20machine%20learning%20techniques%20to%20drive%20and%20grow%20business%20Presentation%201.pdf)
	* [Self-Serve Reporting Platform on Hadoop at LinkedIn](https://cdn.oreillystatic.com/en/assets/1/event/137/Building%20a%20self-serve%20real-time%20reporting%20platform%20at%20LinkedIn%20Presentation%201.pdf)
	* [Privacy-Preserving Analytics and Reporting at LinkedIn](https://engineering.linkedin.com/blog/2019/04/privacy-preserving-analytics-and-reporting-at-linkedin)
	* [Analytics Platform for Tracking Item Availability at Walmart](https://medium.com/walmartlabs/how-we-build-a-robust-analytics-platform-using-spark-kafka-and-cassandra-lambda-architecture-70c2d1bc8981)
	* [Real-Time Analytics for Mobile App Crashes using Apache Pinot at Uber](https://www.uber.com/en-SG/blog/real-time-analytics-for-mobile-app-crashes/)
	* [HALO: Hardware Analytics and Lifecycle Optimization at Facebook](https://code.fb.com/data-center-engineering/hardware-analytics-and-lifecycle-optimization-halo-at-facebook/)
	* [RBEA: Real-time Analytics Platform at King](https://techblog.king.com/rbea-scalable-real-time-analytics-king/)
	* [AresDB: GPU-Powered Real-time Analytics Engine at Uber](https://eng.uber.com/aresdb/)
	* [AthenaX: Streaming Analytics Platform at Uber](https://eng.uber.com/athenax/)
	* [Jupiter: Config Driven Adtech Batch Ingestion Platform at Uber](https://www.uber.com/en-SG/blog/jupiter-batch-ingestion-platform/)
	* [Delta: Data Synchronization and Enrichment Platform at Netflix](https://medium.com/netflix-techblog/delta-a-data-synchronization-and-enrichment-platform-e82c36a79aee)
	* [Keystone: Real-time Stream Processing Platform at Netflix](https://medium.com/netflix-techblog/keystone-real-time-stream-processing-platform-a3ee651812a)
	* [Databook: Turning Big Data into Knowledge with Metadata at Uber](https://eng.uber.com/databook/)
	* [Amundsen: Data Discovery & Metadata Engine at Lyft](https://eng.lyft.com/amundsen-lyfts-data-discovery-metadata-engine-62d27254fbb9)
	* [Maze: Funnel Visualization Platform at Uber](https://eng.uber.com/maze/)
	* [Metacat: Making Big Data Discoverable and Meaningful at Netflix](https://medium.com/netflix-techblog/metacat-making-big-data-discoverable-and-meaningful-at-netflix-56fb36a53520)
	* [SpinalTap: Change Data Capture System at Airbnb](https://medium.com/airbnb-engineering/capturing-data-evolution-in-a-service-oriented-architect


Question: How to: Architecture

Answer:
* [Tech Stack at Medium](https://medium.engineering/the-stack-that-helped-medium-drive-2-6-millennia-of-reading-time-e56801f7c492)
* [Tech Stack at Shopify](https://engineering.shopify.com/blogs/engineering/e-commerce-at-scale-inside-shopifys-tech-stack)
* [Building Services (4 parts) at Airbnb](https://medium.com/airbnb-engineering/building-services-at-airbnb-part-4-23c95e428064)
* [Architecture of Evernote](https://evernote.com/blog/a-digest-of-evernotes-architecture/)
* [Architecture of Chat Service (3 parts) at Riot Games](https://engineering.riotgames.com/news/chat-service-architecture-persistence)
* [Architecture of League of Legends Client Update](https://technology.riotgames.com/news/architecture-league-client-update)
* [Architecture of Ad Platform at Twitter](https://blog.twitter.com/engineering/en_us/topics/infrastructure/2020/building-twitters-ad-platform-architecture-for-the-future.html)
* [Architecture of API Gateway at Uber](https://eng.uber.com/architecture-api-gateway/)
* [Architecture of API Gateway at Tinder](https://medium.com/tinder/how-we-built-the-tinder-api-gateway-831c6ca5ceca)
* [Basic Architecture of Slack](https://slack.engineering/how-slack-built-shared-channels-8d42c895b19f)
* [Lightweight Distributed Architecture to Handle Thousands of Library Releases at eBay](https://tech.ebayinc.com/engineering/a-lightweight-distributed-architecture-to-handle-thousands-of-library-releases-at-ebay/)
* [Back-end at LinkedIn](https://engineering.linkedin.com/architecture/brief-history-scaling-linkedin)
* [Back-end at Flickr](https://yahooeng.tumblr.com/post/157200523046/introducing-tripod-flickrs-backend-refactored)
* [Infrastructure (3 parts) at Zendesk](https://medium.com/zendesk-engineering/the-history-of-infrastructure-at-zendesk-part-3-foundation-team-forming-and-evolving-9859e40f5390)
* [Cloud Infrastructure at Grubhub](https://bytes.grubhub.com/cloud-infrastructure-at-grubhub-94db998a898a)
* [Real-time Presence Platform at LinkedIn](https://engineering.linkedin.com/blog/2018/01/now-you-see-me--now-you-dont--linkedins-real-time-presence-platf)
* [Settings Platform at LinkedIn](https://engineering.linkedin.com/blog/2019/05/building-member-trust-through-a-centralized-and-scalable-setting)
* [Nearline System for Scale and Performance (2 parts) at Glassdoor](https://medium.com/glassdoor-engineering/building-a-nearline-system-for-scale-and-performance-part-ii-9e01bf51b23d)
* [Real-time User Action Counting System for Ads at Pinterest](https://medium.com/@Pinterest_Engineering/building-a-real-time-user-action-counting-system-for-ads-88a60d9c9a)
* [API Platform at Riot Games](https://engineering.riotgames.com/news/riot-games-api-deep-dive)
* [Games Platform at The New York Times](https://open.nytimes.com/play-by-play-moving-the-nyt-games-platform-to-gcp-with-zero-downtime-cf425898d569)
* [Kabootar: Communication Platform at Swiggy](https://bytes.swiggy.com/kabootar-swiggys-communication-platform-e5a43cc25629)
* [Simone: Distributed Simulation Service at Netflix](https://medium.com/netflix-techblog/https-medium-com-netflix-techblog-simone-a-distributed-simulation-service-b2c85131ca1b)
* [Seagull: Distributed System that Helps Running > 20 Million Tests Per Day at Yelp](https://engineeringblog.yelp.com/2017/04/how-yelp-runs-millions-of-tests-every-day.html)
* [PriceAggregator: Intelligent System for Hotel Price Fetching (3 parts) at Agoda](https://medium.com/agoda-engineering/priceaggregator-an-intelligent-system-for-hotel-price-fetching-part-3-52acfc705081)
* [Phoenix: Testing Platform (3 parts) at Tinder](https://medium.com/tinder-engineering/phoenix-tinders-testing-platform-part-iii-520728b9537)
* [Hexagonal Architecture at Netflix](https://netflixtechblog.com/ready-for-changes-with-hexagonal-architecture-b315ec967749)
* [Architecture of Sticker Services at LINE](https://www.slideshare.net/linecorp/architecture-sustaining-line-sticker-services)
* [Stack Overflow Enterprise at Palantir](https://medium.com/@palantir/terraforming-stack-overflow-enterprise-in-aws-47ee431e6be7)
* [Architecture of Following Feed, Interest Feed, and Picked For You at Pinterest](https://medium.com/@Pinterest_Engineering/building-a-dynamic-and-responsive-pinterest-7d410e99f0a9)
* [API Specification Workflow at WeWork](https://engineering.wework.com/our-api-specification-workflow-9337448d6ee6)
* [Media Database at Netflix](https://medium.com/netflix-techblog/implementing-the-netflix-media-database-53b5a840b42a)
* [Member Transaction History Architecture at Walmart](https://medium.com/walmartlabs/member-transaction-history-architecture-8b6e34b87c21)
* [Sync Engine (2 parts) at Dropbox](https://dropbox.tech/infrastructure/-testing-our-new-sync-engine)
* [Ads Pacing Service at Twitter](https://blog.twitter.com/engineering/en_us/topics/infrastructure/2021/how-we-built-twitter-s-highly-reliable-ads-pacing-service)
* [Rapid Event Notification System at Netflix](https://netflixtechblog.com/rapid-event-notification-system-at-net


Question: How to: Organization

Answer:
* [Engineering Levels at SoundCloud](https://developers.soundcloud.com/blog/engineering-levels)
* [Engineering Roles at Palantir](https://medium.com/palantir/dev-versus-delta-demystifying-engineering-roles-at-palantir-ad44c2a6e87)
* [Engineering Career Framework at Dropbox](https://dropbox.tech/culture/our-updated-engineering-career-framework)
* [Scaling Engineering Teams at Twitter](https://www.youtube.com/watch?v=-PXi_7Ld5kU)
* [Scaling Decision-Making Across Teams at LinkedIn](https://engineering.linkedin.com/blog/2018/03/scaling-decision-making-across-teams-within-linkedin-engineering)
* [Scaling Data Science Team at GOJEK](https://blog.gojekengineering.com/the-dynamics-of-scaling-an-organisation-cb96dbe8aecd)
* [Scaling Agile at Zalando](https://engineering.zalando.com/posts/2018/05/scaling-agile-zalando.html)
* [Scaling Agile at bol.com](https://hackernoon.com/how-we-run-bol-com-with-60-autonomous-teams-fe7a98c0759)
* [Lessons Learned from Scaling a Product Team at Intercom](https://blog.intercom.com/how-we-build-software/)
* [Hiring, Managing, and Scaling Engineering Teams at Typeform](https://medium.com/@eleonorazucconi/toby-oliver-cto-typeform-on-hiring-managing-and-scaling-engineering-teams-86bef9e5a708)
* [Scaling the Datagram Team at Instagram](https://instagram-engineering.com/scaling-the-datagram-team-fc67bcf9b721)
* [Scaling the Design Team at Flexport](https://medium.com/flexport-design/designing-a-design-team-a9a066bc48a5)
* [Team Model for Scaling a Design System at Salesforce](https://medium.com/salesforce-ux/the-salesforce-team-model-for-scaling-a-design-system-d89c2a2d404b)
* [Building Analytics Team (4 parts) at Wish](https://medium.com/wish-engineering/scaling-the-analytics-team-at-wish-part-4-recruiting-2a9823b9f5a)
* [From 2 Founders to 1000 Employees at Transferwise](https://medium.com/transferwise-ideas/from-2-founders-to-1000-employees-how-a-small-scale-startup-grew-into-a-global-community-9f26371a551b)
* [Lessons Learned Growing a UX Team from 10 to 170 at Adobe](https://medium.com/thinking-design/lessons-learned-growing-a-ux-team-from-10-to-170-f7b47be02262)
* [Five Lessons from Scaling at Pinterest](https://medium.com/@sarahtavel/five-lessons-from-scaling-pinterest-6a699a889b08)
* [Approach Engineering at Vinted](http://engineering.vinted.com/2018/09/04/how-we-approach-engineering-at-vinted/)
* [Using Metrics to Improve the Development Process (and Coach People) at Indeed](https://engineering.indeedblog.com/blog/2018/10/using-metrics-to-improve-the-development-process-and-coach-people/)
* [Mistakes to Avoid while Creating an Internal Product at Skyscanner](https://medium.com/@SkyscannerEng/9-mistakes-to-avoid-while-creating-an-internal-product-63d579b00b1a)
* [RACI (Responsible, Accountable, Consulted, Informed) at Etsy](https://codeascraft.com/2018/01/04/selecting-a-cloud-provider/)
* [Four Pillars of Leading People (Empathy, Inspiration, Trust, Honesty) at Zalando](https://engineering.zalando.com/posts/2018/10/four-pillars-leadership.html)
* [Pair Programming at Shopify](https://engineering.shopify.com/blogs/engineering/pair-programming-explained)
* [Distributed Responsibility at Asana](https://blog.asana.com/2017/12/distributed-responsibility-engineering-manager/)
* [Rotating Engineers at Zalando](https://engineering.zalando.com/posts/2019/03/rotating-engineers-at-zalando.html)
* [Experiment Idea Review at Pinterest](https://medium.com/pinterest-engineering/how-pinterest-supercharged-its-growth-team-with-experiment-idea-review-fd6571a02fb8)
* [Tech Migrations at Spotify](https://engineering.atspotify.com/2020/06/25/tech-migrations-the-spotify-way/)
* [Improving Code Ownership at Yelp](https://engineeringblog.yelp.com/2021/01/whose-code-is-it-anyway.html)
* [Agile Code Base at eBay](https://tech.ebayinc.com/engineering/how-creating-an-agile-code-base-helped-ebay-pivot-for-apple-silicon/)
* [Agile Data Engineering at Miro](https://medium.com/miro-engineering/agile-data-engineering-at-miro-ec2dcc8a3fcb)
* [Automated Incident Management through Slack at Airbnb](https://medium.com/airbnb-engineering/incident-management-ae863dc5d47f)
* [Refactor Organization at BBC](https://medium.com/bbc-product-technology/refactor-organisation-80e4e171d922)
* [Code Review](https://ai.google/research/pubs/pub47025)
	* [Code Review at Palantir](https://medium.com/@palantir/code-review-best-practices-19e02780015f)
	* [Code Review at LINE](https://engineering.linecorp.com/en/blog/effective-code-review/)
	* [Code Reviews at Medium](https://medium.engineering/code-reviews-at-medium-bed2c0dce13a)
	* [Code Review at LinkedIn](https://engineering.linkedin.com/blog/2018/06/scaling-collective-code-ownership-with-code-reviews)
	* [Code Review at Disney](https://medium.com/disney-streaming/the-secret-to-better-code-reviews-c14c7884b9ac)
	* [Code Review at Netlify](https://www.netlify.com/blog/2020/03/05/feedback-ladders-how-we-encode-code-reviews-at-netlify/)


Question: How to: A Piece of Cake

Answer: Roses are red. Violets are blue. [Binh](https://nguyenquocbinh.org/) likes sweet. [Treat Binh a tiramisu?](https://paypal.me/binhnguyennus) :cake:


Question: How to: WHO WE ARE

Answer:
Netdata is an open-source, real-time infrastructure monitoring platform. Monitor, detect, and act across your entire infrastructure.

**Core Advantages:**

* **Instant Insights** – With Netdata you can access per-second metrics and visualizations.
* **Zero Configuration** – You can deploy immediately without complex setup.
* **ML-Powered** – You can detect anomalies, predict issues, and automate analysis.
* **Efficient** – You can monitor with minimal resource usage and maximum scalability.
* **Secure & Distributed** – You can keep your data local with no central collection needed.

With Netdata, you get real-time, per-second updates. Clear **insights at a glance**, no complexity.
All heroes have a great origin story. Click to discover ours.
In 2013, at the company where Costa Tsaousis was COO, a significant percentage of their cloud-based transactions failed silently, severely impacting business performance.

Costa and his team tried every troubleshooting tool available at the time. None could identify the root cause. As Costa later wrote:

“*I couldn’t believe that monitoring systems provide so few metrics and with such low resolution, scale so badly, and cost so much to run.*”

Frustrated, he decided to build his own monitoring tool, starting from scratch.

That decision led to countless late nights and weekends. It also sparked a fundamental shift in how infrastructure monitoring and troubleshooting are approached, both in method and in cost.


Question: How to: Most Energy-Efficient Monitoring Tool

Answer:
According to the [University of Amsterdam study](https://www.ivanomalavolta.com/files/papers/ICSOC_2023.pdf), Netdata is the most energy-efficient tool for monitoring Docker-based systems. The study also shows Netdata excels in CPU usage, RAM usage, and execution time compared to other monitoring solutions.

---


Question: How to: Key Features

Answer:
| Feature                    | Description                               | What Makes It Unique                                     |
|----------------------------|-------------------------------------------|----------------------------------------------------------|
| **Real-Time**              | Per-second data collection and processing | Works in a beat – click and see results instantly        |
| **Zero-Configuration**     | Automatic detection and discovery         | Auto-discovers everything on the nodes it runs           |
| **ML-Powered**             | Unsupervised anomaly detection            | Trains multiple ML models per metric at the edge         |
| **Long-Term Retention**    | High-performance storage                  | ~0.5 bytes per sample with tiered storage for archiving  |
| **Advanced Visualization** | Rich, interactive dashboards              | Slice and dice data without query language               |
| **Extreme Scalability**    | Native horizontal scaling                 | Parent-Child centralization with multi-million samples/s |
| **Complete Visibility**    | From infrastructure to applications       | Simplifies operations and eliminates silos               |
| **Edge-Based**             | Processing at your premises               | Distributes code instead of centralizing data            |

> [!NOTE]
> Want to put Netdata to the test against Prometheus?
> Explore the [full comparison](https://www.netdata.cloud/blog/netdata-vs-prometheus-2025/).

---


Question: How to: Netdata Ecosystem

Answer:
This three-part architecture enables you to scale from single nodes to complex multi-cloud environments:

| Component         | Description                                                                                | License                                         |
|-------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------|
| **Netdata Agent** | • Core monitoring engine
• Handles collection, storage, ML, alerts, exports
• Runs on servers, cloud, K8s, IoT
• Zero production impact            | [GPL v3+](https://www.gnu.org/licenses/gpl-3.0) |
| **Netdata Cloud** | • Enterprise features
• User management, RBAC, horizontal scaling
• Centralized alerts
• Free community tier
• No metric storage centralization |                                                 |
| **Netdata UI**    | • Dashboards and visualizations
• Free to use
• Included in standard packages
• Latest version via CDN                                             | [NCUL1](https://app.netdata.cloud/LICENSE.txt)  |


Question: How to: What You Can Monitor

Answer:
With Netdata you can monitor all these components across platforms:

|                                                                                Component |              Linux               | FreeBSD | macOS |                      Windows                      |
|------------------------------------------------------------------------------------------------------------:|:--------------------------------:|:-------:|:-----:|:-------------------------------------------------:|
|                             **System Resources**
CPU, Memory and system shared resources
|               Full               |   Yes   |  Yes  |                        Yes                        |
|                                **Storage**
Disks, Mount points, Filesystems, RAID arrays
|               Full               |   Yes   |  Yes  |                        Yes                        |
|                                 **Network**
Network Interfaces, Protocols, Firewall, etc
|               Full               |   Yes   |  Yes  |                        Yes                        |
|                        **Hardware & Sensors**
Fans, Temperatures, Controllers, GPUs, etc
|               Full               |  Some   | Some  |                       Some                        |
|                                       **O/S Services**
Resources, Performance and Status
| Yes
`systemd`
|    -    |   -   |                         -                         |
|                                      **Processes**
Resources, Performance, OOM, and more
|               Yes                |   Yes   |  Yes  |                        Yes                        |
|                                                                             System and Application **Logs** | Yes
`systemd`-journal |    -    |   -   | Yes
`Windows Event Log`, `ETW`
|
|                                 **Network Connections**
Live TCP and UDP sockets per PID
|               Yes                |    -    |   -   |                         -                         |
|                               **Containers**
Docker/containerd, LXC/LXD, Kubernetes, etc
|               Yes                |    -    |   -   |                         -                         |
|                                 **VMs** (from the host)
KVM, qemu, libvirt, Proxmox, etc
| Yes
`cgroups`
|    -    |   -   |         Yes
`Hyper-V`
|
|                       **Synthetic Checks**
Test APIs, TCP ports, Ping, Certificates, etc
|               Yes                |   Yes   |  Yes  |                        Yes                        |
| **Packaged Applications**
nginx, apache, postgres, redis, mongodb,
and hundreds more
|               Yes                |   Yes   |  Yes  |                        Yes                        |
|                              **Cloud Provider Infrastructure**
AWS, GCP, Azure, and more
|               Yes                |   Yes   |  Yes  |                        Yes                        |
|                       **Custom Applications**
OpenMetrics, StatsD and soon OpenTelemetry
|               Yes                |   Yes   |  Yes  |                        Yes                        |

On Linux, you can continuously monitor all kernel features and hardware sensors for errors, including Intel/AMD/Nvidia GPUs, PCI AER, RAM EDAC, IPMI, S.M.A.R.T, Intel RAPL, NVMe, fans, power supplies, and voltage readings.

---


Question: How to: Getting Started

Answer: You can install Netdata on all major operating systems. To begin:


Question: How to: 1. Install Netdata

Answer:
Choose your platform and follow the installation guide:

* [Linux Installation](https://learn.netdata.cloud/docs/installing/one-line-installer-for-all-linux-systems)
* [macOS](https://learn.netdata.cloud/docs/installing/macos)
* [FreeBSD](https://learn.netdata.cloud/docs/installing/freebsd)
* [Windows](https://learn.netdata.cloud/docs/netdata-agent/installation/windows)
* [Docker Guide](/packaging/docker/README.md)
* [Kubernetes Setup](https://learn.netdata.cloud/docs/installation/install-on-specific-environments/kubernetes)

> [!NOTE]
> You can access the Netdata UI at `http://localhost:19999` (or `http://NODE:19999` if remote).


Question: How to: 2. Configure Collectors

Answer:
Netdata auto-discovers most metrics, but you can manually configure some collectors:

* [All collectors](https://learn.netdata.cloud/docs/data-collection/)
* [SNMP monitoring](https://learn.netdata.cloud/docs/data-collection/monitor-anything/networking/snmp)


Question: How to: 3. Configure Alerts

Answer:
You can use hundreds of built-in alerts and integrate with:

`email`, `Slack`, `Telegram`, `PagerDuty`, `Discord`, `Microsoft Teams`, and more.

> [!NOTE]
> Email alerts work by default if there's a configured MTA.


Question: How to: 4. Configure Parents

Answer:
You can centralize dashboards, alerts, and storage with Netdata Parents:

* [Streaming Reference](https://learn.netdata.cloud/docs/streaming/streaming-configuration-reference)

> [!NOTE]
> You can use Netdata Parents for central dashboards, longer retention, and alert configuration.