Airflow is one of the most widely used Schedulers currently in the tech industry. Initially developed at Airbnb, a few years ago it became an Apache foundation project, quickly becoming one of the foundation top projects. It is a direct competitor of other schedulers such as Spotify's Luigi or newer…
These are the steps I took to install Microsoft's cool Gradient Boosted Models library, LightGBM Step 1. Install CUDA I am not going to explain this step because it is easy to find. Step 2. Install Boost sudo apt-get install libboost-all-dev Step 3. Clone LightGBM and build with CUDA enabled…
If you run pyspark and see this error (it happens in scala-shell as well): Error while instantiating ‘org.apache.spark.sql.hive.HiveSessionState’ The solution is easy, yet ridiculous. 1. Create the folder /tmp/hive 2. Give it chmod permissions sudo chmod -R 777 /tmp/hive Found here…
Yesterday I spent a few hours dealing with what I like to call "the edges of StackOverflow". By that I mean those situations in which you are trying to solve a programming problem (mostly a bug) and you have no idea why its happening, and even worse, no amount of…
Let's assume we have a service set up as follows: [Unit] Description=systemd_microservice [Service] User=USER Group=GROUP WorkingDirectory=systemd_working_directory ExecStart=/usr/bin/python python_scripts.py SuccessExitStatus=143 TimeoutStopSec=10 Restart=on-failure RestartSec=10 [Install] WantedBy=multi-user.target And inside python_script.py you have a…