# Turn DevOps Easier

> Tips & Tricks for Smoother Ops


## Posts

- [Puppet Series - Automate deploying Go application with Puppet](https://turndevopseasier.com/puppet-series-automate-deploying-go-application-with-puppet/): This guide shows how to automate deploying a Go application with Puppet, covering release discovery, binary management, systemd services, Nginx configuration, and secure settings. A practical approach for keeping lightweight apps consistently updated across servers.
- [Using PuppetDB to leverage your Puppet advanced features](https://turndevopseasier.com/using-puppetdb-to-leverage-your-puppet-advanced-features/): The article discusses why we need PuppetDB and show the steps of using puppetlabs/puppetdb module to speed up PuppetDB setup
- [Migrating from AWS SQS to RabbitMQ: How We Cut Message Queue Costs by 50%](https://turndevopseasier.com/migrating-from-aws-sqs-to-rabbitmq-how-we-cut-message-queue-costs/): Learn why we migrated from AWS SQS to RabbitMQ and how the move reduced message queue costs by 50% in production.
- [Migrating Graphite from Python to Go for Grafana: How I Reduced Disk I/O and Cut AWS Costs](https://turndevopseasier.com/migrating-graphite-from-python-to-go-for-grafana-how-i-reduced-disk-i-o-and-cut-aws-costs/): This blog shares an experience of migrating Graphite from python-based to golang-based. The switch significantly reduced disk I/O pressure and lowered CPU and memory load, ultimately helping us optimise AWS costs as a result.
- [AWS EBS Cost Optimization: 10 Tips to Reduce Your AWS Bill](https://turndevopseasier.com/aws-ebs-cost-optimization-10-tips-to-reduce-your-aws-bill/): This article shares 10 actionable tips to lower your AWS costs by optimizing EBS usage, along with real-world examples that show when each solution should be applied.
- [Puppet Series - Automating Nagios Setup and Auto-Discovery Using Consul](https://turndevopseasier.com/puppet-series-automating-nagios-setup-and-auto-discovery-using-consul/): This article outlines automating the setup of Nagios and Consul on Ubuntu 24.04 using Puppet, enhancing efficiencies for Host and Service Discovery while reducing manual intervention through configuration management and service discovery tooling.
- [How to Overcome Rsync Issues When Migrating Large Amount of Data Between EBS volumes](https://turndevopseasier.com/how-to-overcome-rsync-issues-when-migrating-large-amount-of-data-between-ebs-volumes/): The blog discusses migrating 1.2TB MySQL data between AWS EBS volumes, detailing challenges faced with rsync failures when copying large data. It offers a solution using the screen command to run rsync in the background efficiently.
- [Automating Nagios Host and Service Discovery with Consul](https://turndevopseasier.com/automating-nagios-host-and-service-discovery-with-consul/): Configuring Nagios for hosts can be labor-intensive and error-prone, especially as host numbers increase. Automating this process using tools like Consul and configuration management systems streamlines host monitoring setup in dynamic environments.
- [Consul Service Discovery: Automate HAProxy Configuration with Consul for MySQL](https://turndevopseasier.com/consul-service-discovery-automate-haproxy-configuration-with-consul-for-mysql/): This article is to show another use case of Consul. In general, we can register a lot of different services (MySQL, Redis, Memcache, our own applications, etc.) to Consul. In this way, we use Consul to track the availability of those services. In the HAProxy context, remember that when configuring HAProxy, we have to manually update the backend config pointing to the servers that we use to process the requests. Backends here could be MySQL, our own applications, or any other services. For example, we use HAProxy for load balancing traffic to MySQL databases. I used to write Set up...
- [Automate SFTP User Creation on Ubuntu 24.04](https://turndevopseasier.com/automate-sftp-user-creation-on-ubuntu-24-04/): This blog explains how to automate SFTP user creation on Ubuntu 24.04 using a Bash script, improving efficiency and accuracy, especially for large user lists while supporting password and SSH key authentication.
- [Automate GitLab Runner Registration and Unregistration](https://turndevopseasier.com/automate-gitlab-runner-registration-and-unregistration/): The article discusses automating GitLab Runner registration and unregistration using a custom Bash script that interacts with the GitLab REST API, highlighting the importance of automation for efficient management of multiple runners across various platforms.
- [Configuring & Managing GitLab Runners with Reusable Templates: My Real-World Workflow](https://turndevopseasier.com/configuring-managing-gitlab-runners-with-reusable-templates-my-real-world-workflow/): The article discusses configuring and managing GitLab CI Runners, emphasizing the benefits of using dedicated runners for efficiency and resource allocation. It provides a guide for installing, registering, and optimizing runners, especially on Linux.
- [Puppet Series - Automate GitLab CE setup](https://turndevopseasier.com/puppet-series-automate-gitlab-ce-setup/): This blog post demonstrates automating the setup of GitLab (CE or EE) using Puppet. It details the prerequisites, how to add relevant modules, configure profiles, roles, and manage parameters with Hiera for efficient GitLab installation and management.
- [Optimize Disk Space: Implement Hourly Rotation with Logrotate on Linux](https://turndevopseasier.com/optimize-disk-space-implement-hourly-rotation-with-logrotate-on-linux/): Efficient log file management on Linux servers is essential to prevent excessive disk space usage and application disruption. Using logrotate with hourly configuration can mitigate these issues effectively.
- [A Guide to Policy-Based Autosigning in Puppet](https://turndevopseasier.com/a-guide-to-policy-based-autosigning-in-puppet/): The blog discusses setting up Puppet autosigning for efficient certificate management in scalable environments. It details types of autosigning, particularly focusing on policy-based methods using scripts for security. By allowing automated certificate signing based on defined attributes, it aims to enhance security while reducing manual intervention in node provisioning on Puppet servers.
- [Enabling Puppet Intermediate CA for Enhanced Security](https://turndevopseasier.com/enabling-puppet-intermediate-ca-for-enhanced-security/): This blog outlines the steps to enable Puppet Intermediate CA for improved security, detailing the prerequisites, configuration processes for both Puppet Server and Agents, and troubleshooting certificate handling.
- [Setting Up Nagios Core with Nginx on Ubuntu 24.04](https://turndevopseasier.com/setting-up-nagios-core-with-nginx-on-ubuntu-24-04/): The guide details the process of setting up Nagios Core with Nginx on an Ubuntu server, highlighting prerequisites, installation, configuration, and testing steps. It aims to aid users facing similar challenges in finding comprehensive documentation.
- [Managing Sensitive Data in Puppet Using Hiera-eyaml](https://turndevopseasier.com/managing-sensitive-data-in-puppet-using-hiera-eyaml/): The blog discusses enhancing Puppet security using Hiera and Hiera-eyaml for storing sensitive data securely. It details installation, key management, configuration, and testing with encrypted data, emphasizing best practices for deployment and data management.
- [Puppet Series - Automate Consul Cluster Setup](https://turndevopseasier.com/puppet-series-automate-consul-cluster-setup/): This blog provides a detailed guide on automating the setup of a Consul cluster using Puppet, highlighting configurations for server and agent nodes, Hiera data management, and testing procedures to ensure successful deployment.
- [Mitigating NXDOMAIN Attacks with AWS Route 53 Strategies](https://turndevopseasier.com/mitigating-nxdomain-attacks-with-aws-route-53-strategies/): This blog discusses NXDOMAIN attacks, specifically focusing on their impact on AWS Route 53, and offers strategies for mitigating risks, such as configuring wildcard records and cost-effective DNS practices to protect against these DDoS threats.
- [Consul Service Discovery: Automate Nginx Upstream updates for dynamic load balancing](https://turndevopseasier.com/consul-service-discovery-automate-nginx-upstream-updates-for-dynamic-load-balancing/): This blog illustrates automating Nginx upstream configuration updates using Hashicorp Consul Service Discovery. It details setting up a Consul server and agents, registering a Python application, and configuring Nginx for dynamic load balancing.
- [Set up High Availablity for MySQL Load Balancing via HAProxy](https://turndevopseasier.com/set-up-high-availablity-for-mysql-load-balancing-via-haproxy/): This blog discusses configuring HAProxy for load balancing MySQL databases, enabling high availability with master/slave setups. It outlines prerequisites, installation, configuration details, testing load balancing, and advanced custom health checks for optimal performance.
- [Puppet Series - Automate MySQL Master-Slave Replication Setup](https://turndevopseasier.com/puppet-series-automate-mysql-master-slave-replication-setup/): This blog post demonstrates setting up MySQL Master/Slave replication using Puppet for automated configuration, emphasizing the use of roles, profiles, and Hiera for managing multiple database nodes efficiently.
- [Optimize AWS Data Transfer Costs with Nginx Gzip Compression](https://turndevopseasier.com/optimize-aws-data-transfer-costs-with-nginx-gzip-compression/): Enabling the Gzip module in Nginx on AWS significantly reduces data transfer size and costs by compressing various MIME types. Proper configuration enhances performance and load times for websites.
- [Avoid unexpected Regional Data Transfer from an FTP server in AWS](https://turndevopseasier.com/avoid-unexpected-regional-data-transfer-from-an-ftp-server-in-aws/): The blog discusses a costly incident involving improper FTP server configuration in AWS, leading to unexpected Regional Data Transfer charges due to public IP usage. Solutions included correcting server configurations to mitigate costs.
- [Integrate SonarQube with GitLab CI/CD for Merge Requests](https://turndevopseasier.com/integrate-sonarqube-with-gitlab-ci-cd-for-merge-requests/): Code review is crucial in software development for early error detection, enhancing security. This blog focuses on utilizing SonarQube with GitLab CI/CD for automated code analysis, ensuring maintainability, reliability, and security during integration.
- [Real use case: Nginx Rate Limit + Fail2ban to prevent massive spam POST requests](https://turndevopseasier.com/real-use-case-nginx-rate-limit-fail2ban-to-prevent-massive-spam-post-requests/): This article discusses using Nginx Rate Limiting and Fail2Ban to prevent spam POST requests and potential server crashes. It details the implementation of rate limiting rules and Fail2Ban configurations to enhance website security effectively.
- [Secure File Transfers: Setting Up SFTP on Ubuntu 24.04](https://turndevopseasier.com/secure-file-transfers-setting-up-sftp-on-ubuntu-24-04/): This blog provides a comprehensive guide on setting up an SFTP server using either password or SSH key authentication on an Ubuntu system.
- [Secure your Nginx sites with Let's Encrypt SSL by automating with Certbot](https://turndevopseasier.com/secure-your-nginx-sites-with-lets-encrypt-ssl-by-automating-with-certbot/): SSL/TLS certificates are an integral part of the modern digital world. This not only establishes your customers’ trust but also secures your websites from cyber threats. However, managing SSL certificates manually can be time-consuming and frustrating. Furthermore, if you’re a small-to-midsize business, the cost of commercial certificates may also be unnecessarily high, while you can easily utilize the power of open-source tools. Actually, even if you work for a big corporation, there might be some cases where you could still leverage these open-source tools. With that being said, in this blog, I want to introduce Let’s Encrypt, a free, automated,...
- [Efficient Puppet Code Deployment using r10k](https://turndevopseasier.com/efficient-puppet-code-deployment-using-r10k/): This blog explores configuring r10k for managing Puppet code efficiently. It outlines prerequisites and steps to set up r10k, allowing easier deployment and version control of Puppet environments.
- [How to Set Up a Local DNS Server with Dnsmasq in Ubuntu 24.04 (Fast & Easy)](https://turndevopseasier.com/how-to-set-up-a-local-dns-server-with-dnsmasq-in-ubuntu-24-04-fast-easy/): Dnsmasq simplifies local DNS setup, providing essential services for small networks, enhancing efficiency, and serving as a backup solution.
- [Mastering Puppet: Implementing Roles and Profiles Effectively In Reality](https://turndevopseasier.com/mastering-puppet-implementing-roles-and-profiles-effectively/): This blog walks you through the Roles and Profiles method in design pattern in Puppet to enhance code reusability and organization, crucial for managing complex infrastructures effectively.
- [Setup Puppet 8 on Ubuntu 24.04 - Configuration Management for a scaling enterprise](https://turndevopseasier.com/setup-puppet-8-on-ubuntu-24-04-configuration-management-for-a-scaling-enterprise/): This article provides a guide to installing and setting up a Puppet server, offering insights on its purpose, advantages over manual configurations, and detailed installation steps for Puppet and OpenVox.
- [Dkron - Distributed Cron System - Solution to replace traditional cron and optimise cloud cost](https://turndevopseasier.com/dkron-distributed-cron-system-solution-to-replace-traditional-cron-and-optimise-cloud-cost/): Dkron and AWS Spot Instances - a perfect combination for you to replace the traditional cron system, improve reliability, scalability, and optimise cloud cost significantly.
- [Centralise logs with Filebeat + Logstash + Elasticsearch + Kibana](https://turndevopseasier.com/centralise-logs-with-filebeat-logstash-elasticsearch-kibana/): This article provides a step-by-step guide to implementing a centralized logging system using the ELK stack (Elasticsearch, Logstash, and Kibana) to efficiently collect, process, and visualize log data from nginx and system source.
- [A CLI tool to delete a bulk AWS snapshots and keep the expected snapshots based on snapshot age](https://turndevopseasier.com/a-cli-tool-to-delete-a-bulk-aws-snapshots-and-keep-the-expected-snapshots-based-on-snapshot-age/): When operating EC2, we might usually run into the situation of removing the old unused snapshots, and when you have e.g. hundreds of old snapshots, this becomes hard to delete manually. Or you probably want to find a way to automate this process when new snapshots become old snapshots after a while. In this post, I’m sharing a CLI script for deleting AWS snapshots in bulk that I’ve written in the Go language. The script deletes snapshots based on snapshot age and the expected number of snapshots we want to keep according to CLI’s supplied options. Presequisites In Linux: or...
- [GitLab CI/CD: Automate build and publish docker image to a private container registry.](https://turndevopseasier.com/gitlab-ci-cd-automate-build-and-publish-docker-image-to-a-private-container-registry/): This blog post details how to automate the building and publishing of custom Docker images to a GitLab Container Registry using CI/CD pipelines, including creating access tokens and configuring necessary files.
- [20 tips to speed up your GitLab CI/CD pipelines in 2023](https://turndevopseasier.com/20-tips-to-speed-up-your-gitlab-ci-cd-pipelines-in-2023/): The article discusses 20 strategies to optimize GitLab CI/CD pipelines, reducing runtime significantly, enhancing efficiency, and improving developer experience.
- [Deploying a Flask app with Gunicorn and Nginx on Ubuntu 20.04](https://turndevopseasier.com/deploying-flask-app-with-gunicorn-and-nginx-on-ubuntu-20-04/): This post details the deployment of a Flask application on an Ubuntu server utilizing Gunicorn and Nginx, covering installation, configuration, and accessing the application via a domain.
- [GitLab - Trigger CI/CD Pipeline run only when a merge request is approved](https://turndevopseasier.com/gitlab-trigger-ci-cd-pipeline-run-only-when-a-merge-request-is-approved/): This guide outlines how to trigger a GitLab CI/CD pipeline immediately after a merge request approval using a Webhook and custom script. It details the setup of access tokens, Webhooks, and pipelines through Python and Flask.
- [Part 2: Set up a repository by using Reprepro and make the repository public with AWS S3](https://turndevopseasier.com/part-2-set-up-a-repository-by-using-reprepro-and-make-the-repository-public-with-aws-s3/): Continue from Part 1: How to build a custom Debian repository on AWS S3 with Ubuntu 20.04 and reprepo Set up a repository by using Reprepro Reprepro was created by R. Bernhard who used to “produce, manage and sync a local repository of Debian packages” (also known as Mirrorer). This tool makes us build a custom repository easier and it’s under the GNU licence and completely open source. Step 1 – Reprepro installation Run this command to install Create a dedicated folder for this repository Create a configuration directory Create a distributions config file. Edit file by nano command: And add...
- [How to build a custom Debian repository on AWS S3 with Ubuntu 20.04 and reprepo](https://turndevopseasier.com/how-to-build-a-custom-debian-repository-on-aws-s3-with-ubuntu-20-04-and-reprepo/): Introduction When it comes to installing a program on Ubuntu or other Debian-based Linux distros – what should we do normally? Most of us just run e.g. apt-get install ntp to get the package installed. It may seem like magic, but it’s not. The package manager apt-get is responsible for searching, downloading, and installing packages for you. This process is incredibly convenient. However, what if apt-get cannot locate the program you require in its default repositories? Fortunately, apt-get permits users to define custom download sources known as repositories. In this blog, we will find out how to set up your...
- [Publish a Composer package by using CI/CD](https://turndevopseasier.com/publish-a-composer-package-by-using-ci-cd/): The GitLab Package Registry allows publishing and sharing packages. This guide outlines steps for publishing a PHP project as a Composer package using CI/CD, including creating tokens and configuring .gitlab-ci.yml.
- [How to configure FTP on AWS EC2](https://turndevopseasier.com/how-to-configure-ftp-on-aws-ec2/): This content provides a step-by-step guide on setting up an FTP server using vsftpd on an AWS EC2 instance running Ubuntu 20.04. Key configurations and user creation are detailed for functionality.
- [How to configure VSFTPD server behind a firewall for handling internal and external IPs](https://turndevopseasier.com/how-to-set-up-vsftpd-server-behind-a-firewall-for-handling-internal-and-external-ips/): By default, your EC2 instance normally has an external IP address for connections coming from the public, and an internal IP address for connections coming from your internal network. The internal network is normally for other servers that want to connect to your FTP servers locally without going through the external IP address. When it comes to AWS EC2, data transfer within a local VPC should be free, but when you access your FTP server through the external network, it really costs you money if you transfer a lot of data day by day. The idea of this configuration allows...
- [How to run composer install only if composer.json and composer.lock are are changed](https://turndevopseasier.com/run-composer-install-when-composer-files-are-changed/): If you are a developer, system engineer, or DevOps engineer who works with PHP projects, you know that running composer install is an important task. However, when it comes to continuous integration and continuous deployment (CI/CD), running composer install every time can be time-consuming and unnecessary if the composer files haven’t changed. In this post, I want to share a little trick on how we can automatically run Composer only when the composer.json and composer.lock files have been changed. We can use sha1sum command on Linux to check the SHA1 message digest of the 2 files above if they have...

## Pages

- [Client Portal](https://turndevopseasier.com/clients/): [jetpackcrm_clientportal]
- [Puppet Series](https://turndevopseasier.com/puppet-series/): The blog summaries encompass Puppet Development, covering installation, configuration, efficient code deployment, data management, security, and practical automation use cases for MySQL and Consul setups.
- [https://www.facebook.com/turndevopseasier](https://turndevopseasier.com/https-www-facebook-com-turndevopseasier/)
- [https://gitlab.com/binhdt2611](https://turndevopseasier.com/https-gitlab-com-binhdt2611/)
- [https://www.pinterest.com/](https://turndevopseasier.com/https-www-pinterest-com/)
- [https://www.google.com/](https://turndevopseasier.com/https-www-google-com/)
- [About Me](https://turndevopseasier.com/about/): Welcome to my blog, I’m Binh, currently working as a Systems / DevOps Engineer. Honestly, the title doesn’t matter much to me as I have been working in IT for 10 years so far, not too long, but not new anymore. What I’ve observed over my career is that, regardless of the titles you hold, such as Systems Engineer, DevOps Engineer, Platform Engineer, or Site Reliability Engineer, I personally think they are just puzzle words that employers use to find a suitable person for a position. So, whatever the title is, I think there is still domain knowledge and technologies...


[comment]: # (Generated by Hostinger Tools Plugin)