As quoted by the final source where I got the code from -

Explanation to Rule 26 of the Income Tax Rules, 1962 advises using telegraphic transfer rates of SBI as reference for calculating foreign income or capital gains. SBI publishes the rates daily on its website, barring Sundays and bank holidays. Unfortunately, there is no official way to access historical data.

The Challenge

Scraping a PDF normally is easy, considering that there are already many open-source projects available for the same. But here, this PDF had data present in tables as shown below -

My Approach

Before I found the complete solution to my problem readymade (scroll directly below if you want to skip this), I had made up my mind to build the entire scraper using Node.

I came across many NPM packages to help me with the same, but almost none of them had a built-in approach for parsing tables in a PDF. I struck a gold mine by finally finding an NPM package - pomgui/pdf-tables-parser.

It was the exact solution I was looking for, except, as of 10th January 2023 an update to one of its dependencies pdfjs-dist, the generic build of pdfjs caused the package to break.

The Fix

I would be lying if I said that I fixed the issue in minutes. At first, I thought it was some issue with the node_modules I downloaded and spent some time installing the dependencies I was shown was missing. Only after some time when I went through the source code of the pdf-tables-parser, did I realize the path for the build of pdfjs-dist had changed from pdfjs-dist/es5/build/pdf to pdfjs-dist/build/pdf.

A quick fix and update to that had fixed the missing dependencies errors. I put in a pull request on the GitHub repository of the package.

The Code

1. Downloading the PDF from the website - Using the async await will ensure we can wait for the download asynchronously. The npm package download uses a function to pass in the URL and the output folder path.

    const download = require('download');
    async function fileDownload() {
      // download(link, output folder)
      await download('https://www.sbi.co.in/documents/16012/1400784/FOREX_CARD_RATES.pdf', 'input');
    }
   

2. Converting the acquired PDF to JSON - This was made easy after using the pdf-tables-parser package.

   The code below gets the downloaded PDF from the input folder, extracts the data, and writes it to a report.json file in the out directory.

    const { PdfDocument } = require('@pomgui/pdf-tables-parser');
    const fs = require('fs');
   
    async function PDFToJSON() {
      const pdf = new PdfDocument();
      // pdf.load('path/to/PDF')
      await pdf.load('input/FOREX_CARD_RATES.pdf')
        .then(() => fs.writeFileSync('out/report.json', JSON.stringify(pdf, null, 2), 'utf8'))
        .catch((err) => console.error(err));
    }
   

   Here are the dependencies in the package.json for reference -

    "dependencies": {
        "@pomgui/pdf-tables-parser": "^0.1.0",
        "download": "^8.0.0",
        "path": "^0.12.7",
        "pdfjs": "^2.4.7",
        "pdfjs-dist": "^3.2.146"
      }
   

   The Issue

   This code was inefficient as the JSON was not uniform. No doubt that all the contents of the PDF are scraped but it is too ugly and variable to look at.

   Here is how the JSON looked

    {
              "tableNumber": 7,
              "numrows": 1,
              "numcols": 1,
              "data": [
                [
                  "UNITED STATES DOLLAR   USD/INR  "
                ]
              ]
            },
            {
              "tableNumber": 8,
              "numrows": 2,
              "numcols": 9,
              "data": [
                [
                  "81.25",
                  "  82.75",
                  "  81.18",
                  "  82.92",
                  "  80.5",
                  "  83.2",
                  "  80.2",
                  "  83.4",
                  "  81.12"
                ],
                [
                  "EURO   EUR/INR  "
                ]
              ]
            },
   

   The Python Fix

   This amazing GitHub repository solved my entire issue of scraping the said PDF - https://github.com/sahilgupta/sbi_forex_rates

   Thanks to Sahil's solution the task of scraping had been achieved, leaving me with modifying the existing code to scrape page 1 and upload it to Amazon S3 Bucket.

3. Connecting to the S3 bucket

    import os
    import boto3
    from dotenv import load_dotenv
   
    load_dotenv()
   
    s3 = boto3.client(
        "s3",
        aws_access_key_id=os.getenv("aws_access_key_id"),
        aws_secret_access_key=os.getenv("aws_secret_access_key"),
    )
   

4. Uploading the CSV files

    import pandas as pd
   
    def upload_csv():
        # This loop ensures that all the files and folders in the
        # directory "csv_files" is uploaded
        for path, subdirs, files in os.walk("csv_files"):
            for name in files:
                hc = pd.read_csv(os.path.join(path, name))
                csv_buf = StringIO()
                hc.to_csv(csv_buf, header=True, index=False)
                csv_buf.seek(0)
                s3.put_object(
                    Bucket=os.getenv("bucket_name"),
                    Body=csv_buf.getvalue(),
                    Key=os.path.join(path, name),
                )
        print("Uploaded CSV to bucket")
   

5. Uploading the PDF files

    # This loop ensures that all the files and folders in the
    # directory "pdf_files" is uploaded
    for path, subdirs, files in os.walk("pdf_files"):
            for name in files:
                s3.put_object(
                    Bucket=os.getenv("bucket_name"),
                    Body=os.path.join(path, name),
                    Key=os.path.join(path, name),
                )
        print("Uploaded PDF to bucket")
   

6. Scraping Page 1 - This is simple, since both the pages are the same all you need is to start from the second index for the table headers and start at the 3rd index for getting the values. Hence, the changes to be made to the dump_data function in the source code will be -

    header_row = lines[2]
   

   and

    for line in lines[3:]:
   

Concluding Notes

Scraping using Python was an easier task compared with NodeJs. Maybe my approach was wrong when scraping with Node but I felt that scraping with Python felt cleaner and the work got done faster.

You can now schedule a cron job to run the main function/ file to scrape the SBI Forex website. If a change is detected, it will automatically update the CSV.

What Motivated Me To Build It?

Short answer - The pandemic.

Long answer - September 26th, 2020 marks my first commit to the GitHub repository which hosts the source code to Rufus. It's been 2 years.

Back then I had just gotten started with NodeJs and had a basic idea of what JavaScript was. Discord was fairly popular and was gaining attention due to the pandemic. Everyone was hopping onto Discord to play games together or live stream movies with their loved ones.

I started noticing some issues with the platform back then. It simply was not user-friendly at least at first. If you're totally new to Discord without any exposure to any other similar IRC-type applications you'd have found it overwhelming too. Bots back then had no slash commands. Each bot had its own custom prefix.

Here is where I started. I started building a guidebook for those new to Discord. Simply call Rufus with the appropriate command and it would guide you. Imagine this to be the Bitwarden of all passwords, remember one prefix/command, and conveniently view the prefixes/commands of other bots. Convenient right?

The Open Source Way

With every commit to the source code, the bot started taking shape, new commands started getting added, and more people requested me to add features. The motivation to complete this bot was surreal. I'm not joking when I say this but the constant support from online friends as well as other developers I met during the journey was something I hadn't witnessed before.

A few months after the initial commit I received my first pull request on the repository which felt good because people were beginning to notice the project. From code cleanups to restructuring the code and making it better the open source at GitHub helped mold the project to what it is today.

The bot started gaining stars and I started meeting new developers who built such bots too. It would be useless if I don't mention Dev Shah with whom I had a great time collaborating and writing code. This is someone who at a young age had some really good hold on programming.

Ending Note

Thank you for being a part of this journey. Like every good thing, the chapter of Rufus has come to an end. It is time for me to move ahead. From the 28th of November, Rufus will go dark. There are no plans to rewrite the bot to match the latest Discord API nor are there plans to migrate servers to another hosting service. Nothing can possibly beat the convenience of the free service Heroku offered developers.

If you have read this till here, presenting a list of websites that provide free hosting below. Be free to test them out and then settle on which one can scale up to the project you have.

You can find the GitHub repository for the bot here - Rufus Discord Bot

Sites where you can get free hosting (Source) -

• Netlify.com
• Firebase
• Vercel.com
• GitHubPages
• Surge.sh
• Render.com
• Amazone Web Hosting
• Google Cloud Hosting
• Glitch.com
• Fleek.co
• Begin.com
• Cloudaccess.net
• Infinityfree.net

In This Tutorial

We will be using proxy.py to setup a proxy server to cache and serve content from the internet.

Caching is the process of storing the content of a website in a local cache to speed up the loading of the website.

Imagine you have a lot of devices on the same network. Now if one device visits a website and the website is slow to load, without a proxy all the other devices will have to wait for the website to load.

With a proxy this issue can be minimized due to the caching of the website. The proxy server coordinates with the source server to cache documents such as files, images and web pages.

Prerequisites

⚠️ If you are using a Mac system, the Docker image is broken due to incompatibilities with vpnkit. However you can use Homebrew to install it.

Having a Raspberry Pi is not a requirement for this tutorial. Although if you have a Raspberry Pi you can use it to setup a proxy server.

We will be using Docker to setup the proxy server.

Install Docker from here - Docker Engine

One command is all it will take to setup this proxy.

docker run -d --restart unless-stopped -p 8899:8899 abhinavsingh/proxy.py:latest --plugins proxy.plugin.CacheResponsesPlugin --hostname 0.0.0.0

Explanation

• docker run - Starts the proxy server.
• -d - Runs the proxy server in the background aka detached mode.
• --restart unless-stopped - Restarts the proxy server if it is stopped automatically, however if stopped automatically it will stop.
• -p 8899:8899 - Sets the port to 8899 and expose the internal port of the Docker container to the internal network.
• abhinavsingh/proxy.py:latest - The proxy.py image.
• --plugins proxy.plugin.CacheResponsesPlugin - The plugin to cache responses as mentioned in the documentation here.
• --hostname 0.0.0.0 - Sets the hostname to 0.0.0.0, which means that the proxy server will be accessible from any device on the network. This is important.

💡If you want to check out more about Docker commands, you can check out my article here.

Your proxy is now running on port 8899.

That's about it! You can set your device (recommended) or browser to use the proxy.

Setting It Up

Set your proxy IP to the IP address of the device the docker container is running on.

If you are running the container on a Mac or Linux system use ifconfig to get the IP address of your system.

If its Windows then ipconfig will print the IP address.

⚠️ Most of the times the DHCP server on your router should assign your device a static internal IP address, however if it doesn't, change your device settings from DHCP assignment to static assignment.

Linux

If you have a Linux system you will be finding your proxy settings in Network settings.

Since I use a Debian based distro I have it in Settings > Network > Proxy.

Windows

Settings > Network and Internet > Proxy

Final Notes

You can lookup how this proxy server can be used to connect to multiple devices on the same network. This was a tutorial for setting it up on one device.

If your router supports it, configure your router settings to use the proxy.

Getting Into It

Mind you, I am not the author of this code but just for the sake of documenting everything new I find, I am writing this down. This code has been picked up from a StackOverflow thread (I do not have the link of the original thread but if I do come across it again I will make sure to link it)

The code -

function HTMLImporter() {}

HTMLImporter.import = function (url) {
  var error, http_request, load, script;

  script =
    document.currentScript || document.scripts[document.scripts.length - 1];

  load = function (event) {
    var attribute, index, index1, new_script, old_script, scripts, wrapper;

    wrapper = document.createElement("div");
    wrapper.innerHTML = this.responseText;

    scripts = wrapper.getElementsByTagName("SCRIPT");

    for (index = scripts.length - 1; index > -1; --index) {
      old_script = scripts[index];

      new_script = document.createElement("script");
      new_script.innerHTML = old_script.innerHTML;

      for (index1 = old_script.attributes.length - 1; index1 > -1; --index1) {
        attribute = old_script.attributes[index1];
        new_script.setAttribute(attribute.name, attribute.value);
      }

      old_script.parentNode.replaceChild(new_script, old_script);
    }

    while (wrapper.firstChild) {
      script.parentNode.insertBefore(
        wrapper.removeChild(wrapper.firstChild),
        script
      );
    }

    script.parentNode.removeChild(script);

    this.removeEventListener("error", error);
    this.removeEventListener("load", load);
  };

  error = function (event) {
    this.removeEventListener("error", error);
    this.removeEventListener("load", load);

    alert("there was an error!");
  };

  http_request = new XMLHttpRequest();
  http_request.addEventListener("error", error);
  http_request.addEventListener("load", load);
  http_request.open("GET", url);
  http_request.send();
};

How to use it

Import the code into your main HTML file by adding the whole JS code into a file and using the script tag to do so.

Important - Add the import script at the top before you import your components

After importing the importer code use the following to easily integrate the component into your file -

<script>HTMLImporter.import("./path/to/file")</script>

That is it!! You can now split your HTML file into multiple files and yes this works for static websites too!

Installation

This software is available on Mac OSX, Linux and Windows. Just simply download the installation file from their official site. At the time of writing this was their official site.

For Linux

Since I used Ubuntu my guide will be based on Debian based systems. Just download the Debian/Ubuntu .deb package from the website.

Run the command -

sudo dpkg -i VeraCrypt_File_Name.deb

Replace VeraCrypt_File_Name.deb with the name of your downloaded .deb file.

For Windows

A simple .exe installation will work like a charm.

Setup

• Get your USB stick ready and plug it into your PC.
• Boot up VeraCrypt and click on the Create Volume option.

Two options will load:

1) Create an encrypted file container: If this option is used VeraCrypt will create a file which will be used as a virtual mount point. A key point to remember here is that once a size is allocated to the file it cannot be increased nor decreased as opposed to the next option.

2) Create a non-system partition/drive: If this option is used VeraCrypt will encrypt your whole drive. The size of the drive is the size that will be available to you.

VeraCrypt suggests option 1 but I would say go for 2 if you do not know how much content you plan to encrypt. In this guide we will be going ahead with option 2.

💡 Note - If you are interested in option 1: Check out this video.

• After selecting your encryption type, it is time to select volume type. The options here are quite understandable. Unless you are possessing very sensitive material and your life is at stake go for the Standard VeraCrypt Volume.

After selecting your drive aka the USB stick mount location, there are 2 cases here:

1) Your USB stick may be having content already. 2) Your USB stick is empty.

• If your USB stick is empty no worries, in Volume Creation Mode select the Create encrypted volume and format it option. If your USB stick has content that you want then select the Encrypt partition in place option.

• In Encryption Options if you know what you are doing change it to your preference. I left it at the default options as that sufficed me.

• Volume Password should be something that is strong to avoid dictionary attacks. Usually a unique alphanumeric password should suffice.
• 💡 This is important, if you plan to use your USB stick cross-platform, which you will, it is important that in Format Options you should select NTFS. Users who encrypted their partition and did not format it may not get this option.

Final steps

• After all of this is done you will get a interesting window called the Volume Format. This will ask you to shake your mouse till a particular bar is full.
• Once done a 'Format' option should be available. Click that and depending on the size of your drive and the speed it could take time. My 64GB USB stick took up-to an hour to encrypt.

On a conclusive note to the setup I would like to say the Final steps for those who had content in their volume could be different nevertheless, it isn't rocket science to figure it out.

General Use

To now use your volume after it has been encrypted, ensure the device you want to use your volume on has VeraCrypt.

• Open VeraCrypt and click on Select Device and select the encrypted volume mount point.
• Select the slot you want it to mount on. On Linux it will show as 1,2,3...etc and on Windows it will show as A,B,C...etc.
• Click on Mount after selecting the slot and enter your password. Done! Now just navigate to the mount spot and enjoy accessing your files like a normal volume. Once done click on Dismount.

Videos Used As Reference:

1) How To Use VeraCrypt On Linux

2) Linux Tips - Encrypt USB Drive on Windows Linux and MacOS (VeraCrypt)

3) Hide & Encrypt Your Secret Files With VeraCrypt

Python Code

The camera will keep detecting the bot and track it, when the bot reaches its sticker the camera should run a post request in Python.

The script for the python script is as follows -

import requests
endpoint = 'http://localhost:3000/send-mqtt'
data = {
    'message': 'Right'
}
r = requests.post(url=endpoint, data=data)

If one is connecting via another device they have to simply replace localhost with the IP address of the device the server is being run on.

IP of the server can be found by typing in ipconfig in Windows and scrolling to the IPv4 section.

Note - If the script gives an error saying the module requests is not found just run pip install requests.

The Server

The broker used for sending messages is mosquitto. The url is as follows: mqtt://test.mosquitto.org. The following server is open to anyone hence sensitive messages should be avoided.

Using NodeJs

The first steps involve installing Node itself from their website here.

After Node is setup in a new folder run the following line of code: npm init

Fill in all the details or just keep hitting enter till a package.json file is formed.

Proceed to install the express, body-parser and mqtt packages by using the following command: npm i express body-parser mqtt

If somehow this command fails install each separately in the following format: npm i <package name>

Server Code

var express = require("express");
var bodyParser = require("body-parser");
var app = express();
var mqttHandler = require("./mqtt_handler");

app.use(express.json());
app.use(express.urlencoded({ extended: true }));

var mqttClient = new mqttHandler();
mqttClient.connect();

// Routing
app.post("/send-mqtt", function (req, res) {
  mqttClient.sendMessage(req.body.message);
  res.status(200).send("Message sent to mqtt");
});

var server = app.listen(3000, function () {
  console.log("App running on port.", server.address().port);
});

MQTT Handler

const mqtt = require("mqtt");

class MqttHandler {
  constructor() {
    this.mqttClient = null;
    this.host = "mqtt://test.mosquitto.org";
    this.username = "YOUR_USER"; // mqtt credentials if these are needed to connect
    this.password = "YOUR_PASSWORD";
  }

  connect() {
    // Connect mqtt with credentials (in case of needed, otherwise we can omit 2nd param)
    this.mqttClient = mqtt.connect(this.host);
    // , { username: this.username, password: this.password }
    // Mqtt error calback
    this.mqttClient.on("error", (err) => {
      console.log("An error has occured: ", err);
      this.mqttClient.end();
    });

    // Connection on success
    this.mqttClient.on("connect", () => {
      console.log(`Mqtt client connected`);
    });

    // Mqtt subscriptions
    this.mqttClient.subscribe("mytopic", { qos: 0 });

    // Logging a message when it arives
    this.mqttClient.on("message", function (topic, message) {
      if (!message.toString().startsWith("32333")) {
        console.log(message.toString());
      }
    });

    this.mqttClient.on("close", () => {
      console.log(`mqtt client disconnected`);
    });
  }

  // Sends a mqtt message to topic: mytopic
  sendMessage(message) {
    this.mqttClient.publish("mytopic", message);
  }
}

module.exports = MqttHandler;

Getting Started

The starting few steps can be found here on their website

Starting the mongodb server -

sudo service mongod start

Checking the status of the server -

sudo systemctl status mongod

Stopping the mongodb server -

sudo service mongod stop

Error faced

On trying to get the server status I always got the following error - Active: failed (Result: exit-code)

I tried all the troubleshooting guides until I stumbled on a StackOverflow answer detailing the solution to this Just do those two commands for temporary solution:

sudo rm -rf /tmp/mongodb-27017.sock
sudo service mongod start

Cause of the error

For details:

That shall be fault due to user permissions in .sock file, You may have to change the owner to monogdb user.

chown -R mongodb:mongodb /var/lib/mongodb
chown mongodb:mongodb /tmp/mongodb-27017.sock

Uninstall MongoDB completely

sudo service mongod stop
sudo apt-get purge mongodb-org*
sudo rm -r /var/log/mongodb
sudo rm -r /var/lib/mongodb

First Step

As of 2022 in April the latest version of Xilinx is 14.7. Head over here and click on 14.7, scroll down to the ISE Design Suite and install the Full Installer for Linux. A simple popup to sign-up/in will come up.

Extract the contents

After the download is complete extract the file using the following command -

tar -xvf Xilinx_ISE_DS_Lin_14.7_1015_1.tar

This extracts the whole tar file.

Running The Setup

To further setup the Xilinx software we run the following -

sudo ./xsetup

This will run the setup!

Error fixing

During this you may run into an issue which says libncurses.so.5: cannot open shared object file: No such file or directory. A fix to this is done by running

sudo apt-get install libncurses5

Next Steps

The next steps are intuitive and can easily be figured out by clicking Next

Finally Running It

Run this command first -

source /opt/Xilinx/14.7/ISE_DS/settings64.sh

and then just type

ise

in the command line. You are good to go!

Error I faced

The video mentioned above has a step involving adding the source.. command to .bashrc. This led to errors as well as problems on my bash terminal. So avoid it and follow the step mentioned above.

Pre-requisites

1. Install Spotify for your Linux distro by going to the official site of Spotify and not installing the package using Snap but using the Debian package installation.
2. Download Git from here.
3. Make may already be installed on your device but you can install it using sudo apt-get install build-essential to be on the safe side.
4. Install Rust using the following command - curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

That's about it!

Installation

To finally install the adblocked version run the following commands on your terminal -

git clone https://github.com/abba23/spotify-adblock.git
cd spotify-adblock
make

Finally run - sudo make install

To test if it works run this command - LD_PRELOAD=/usr/local/lib/spotify-adblock.so spotify

This should now launch Spotify with the adblocked version!

Creating a desktop file

If you run Spotify installed by you, you will notice the ads aren't blocked unless you run the command above on the terminal. To solve this issue a small hack exists.

• Fire up your terminal and change your directory on the terminal - cd ~/.local/share/applications
• You now need to create a desktop file in order to get the application in your menu. Create the file and edit it by using - nano spotify-adblock.desktop
• In the editing space copy paste the following code:

    [Desktop Entry]
    Type=Application
    Name=Spotify (adblock)
    GenericName=Music Player
    Icon=spotify-client
    TryExec=spotify
    Exec=env LD_PRELOAD=/usr/local/lib/spotify-adblock.so spotify %U
    Terminal=false
    MimeType=x-scheme-handler/spotify;
    Categories=Audio;Music;Player;AudioVideo;
    StartupWMClass=spotify
  

• Ctrl + X, Y and hit enter to exit and save the file!

Done! Now head over to your application menu and you will be seeing a new desktop icon called Spotify (adblock)

Caveats

1. The app does not minimize to dock even if you enable the setting instead it will just shut down. Hence you will have to minimize it and keep it in your app tray.
2. Apart from ad blocking this mod does not offer any other premium features like downloading. (unlimited skips are automatically enabled in this mod)

Quick Note - Check out Pop!_OS if you are still deciding distros. Thank me later!

VMware

Install the VMware Workstation first from the website (one can optionally get the keys from GitHub and get the Pro version for free).

Download the Ubuntu iso (in my case Windows) on the host machine which can be found on the official Ubuntu site.

Open VMware and create new VM followed by these steps:

• Will install OS later
• Linux - Ubuntu 64bit
• Name the VM whatever you like
• Give it at least 40GB and it stays in split
• Customize Hardware:- 4GB RAM minimum, Cores - 4, New CD/DVD => Select iso image, Finish
• Play VM => Software update download and install
• Install Ubuntu
• Minimal Install (If preferred you can go for the other one but I did not require that)
• Check the "Install third party checkbox"
• Erase and install Ubuntu (don't worry your data on your host PC will not be affected if this option is clicked)

Rest steps are easy and configurable on your own.

Let it install, it takes some time and on completion a success message is shown. Once that is done a prompt to restart the VM will be shown and you are done!

To sync Ubuntu's calendar with your Google Calendar you can login with Google too!

Some Fixes I had To Search For

1. In the minimalist version the codecs for audio was missing so you have to install ffmpeg using sudo from the CLI.
2. To activate the camera from VMware, when on full screen mode hover to the top and find the camera icon near the notes and right click on it. This will present a option to Enable the camera. Wait for some time and if it gives an error try again. On your second or third try it should work after giving a prompt.
3. Use the update command to find for updates and upgrade to upgrade the packages.

Dual Boot (Windows)

Some pre-requisites for doing this include -

• A good USB flash drive (4 GB minimum to install)
• Balena Etcher installed on Windows or Ventoy (recommended)
• The latest Ubuntu .iso file downloaded and ready

Although the above method mentions Balena Etcher, I feel that should be substituted with Ventoy.

Once you have the above mentioned items ready all you have to do is fire up Etcher with the USB plugged in to your device. Etcher will automatically detect your USB. On rare circumstances if Etcher does not detect your USB, you should check if the USB is being detected by your device first.

A FLASH option should appear when everything is ready. Flashing takes some time and once done you are good to go!

Head over to Windows and search for Disk Management and create a new partition to install Ubuntu in. A minimum of 40GB is recommended, although I had allocated 100GB.

Note - The steps mentioned below are for Dell devices, your BIOS menu and settings may be different so do refer their documentations if the need be. Although the steps mentioned below shouldn't defer depending on device.

The following steps have to be done in order to dual boot Ubuntu -

• Reboot your device and when it is just about to startup press F12 or whichever shortcut takes you into BIOS mode.
• Once into BIOS mode, in Boot Menu or in Boot Devices you will find your USB device name mentioned. Click on that and wait for the screen to load.
• Once the Ubuntu screen loads click on Install Ubuntu.
• Select Normal Installation and you can then optionally install third-party software for graphics etc.
• This step is the most important step of all. When the option of Installation Type is asked, click on Something else, find for your partition allocated for Ubuntu in the GUI. In the popup select Primary as the type of partition, set the use as Ext4 journaling file system and finally the mount point should be set as /. Verify if the Windows Boot Manager has the type efi and then go ahead and click on Install Now.
• The next few steps are self-explanatory and the installation should now take place.

Now on rebooting your device you shall notice that a bootloader called GRUB starts up. On selecting Ubuntu, Ubuntu shall load and on selecting Windows Boot Manager Windows will load.

Issues Faced In Dual Boot

One major issue faced when installing Ubuntu on Dell was that I had to turn off RST.

This article provides a great solution to fix the issue. However if you lazy to read the article here are the steps to fix the issue -

• Open CMD in Windows (Run as administrator) and type in bcdedit /set {current} safeboot minimal
• Go into BIOS setup and search for SATA mode and change the current setting from whatever it was to AHCI. A frightening question can be asked: All existing data stored on the drives will be erased when resetting the controller mode. Do you want to proceed? [Yes] [No] just click on Yes without worrying. Press F10 and it will save and exit from BIOS Setup.
• The computer will startup in safe boot mode. Open CMD and type the following - bcdedit /deletevalue {current} safeboot

And done! Your Ubuntu should start without any issues now!

Personal Setup

• Spotify without ads :)
• Grub themes.
• KDE Connect to connect it with your phone.
• Gnome Tweaks

What Was The Idea?

The idea was to replace Windows 7 with a more secure OS as upgrading the Windows version would have been difficult.

Which OS?

To select the OS at first I decided to go with Zorin lite which has 32 bit support. After flashing my USB and booting it up, everything functioned perfectly until the install button was clicked.

After nearly 4 tries and even being connected on a reasonable 50Mbps speed the installation did not continue and the installer always got hung at the Install updates while installing Zorin screen.

This led me to finally conclude that Zorin OS may not be compatible with the PC I have.

After a little research and finding I struck gold when I found a good alternative to the Windows 7 environment - Q4OS. At first I was skeptical about this too but it worked wonders.

About Q4OS's Installation Process

It started with a blank blue screen which did scare me at first. Later the installation had 2 options Manual and Automatic. As I always have believed that manual is always preferable as one gets to chose what they require and don't require.

After selecting manual, a few more steps later and after around 10 minutes I managed to clean the disk that had Windows on it and substitute it with Q4OS.

There would be some issues during the installation for example the screen may just get frozen at a blue screen but after a few minutes the installation resumes normally.

Conclusion

In conclusion Q4OS is a good alternative for PC's that cannot run Windows higher than 7. This OS can be used to bring old PC's back to life. Lastly Q4OS is an operating system based on Debian.

Getting Started

Ensure you download the .iso file from their website. Scroll down to the download section, if you wish to pay you can pay, else click on custom and enter the amount as 0. Once that is done the download will be initiated.

The Installation

The same instructions are mentioned on their installation guide over here. Be sure to check the specifications required for the USB flash drive as well as the system.

Flashing

Once the image file is downloaded Etcher does the magic. The process takes some time so hold tight.

Booting

The device used to install Elementary in my case was Dell hence on booting F12 was enough to enter Boot Mode. Once you are in boot mode the USB should be visible in the options on the left (varies per device). Click on it and voila!

• If you are booting from the EOS image for the first time let it run a file check. Once the check is done a basic setup screen will pop-up. Select the clear drive option and wait for some time.
• Optionally the demo mode can be tried on first to check how your device suits with EOS.
• After the initial setup is done and an account is setup you are ready to go!!

NOW THE NEXT TIME YOU POWER ON YOUR DEVICE, IT SHOULD AUTOMATICALLY START EOS!

(Side note - You can eject the USB and remove it now)

Initial tips and tricks

• Firefox or your preferred browser can be installed from FlatHub. All major apps can be found there.
• The weird close and maximize button can be tweaked too using this Pantheon Tweaks package
• In settings I noticed the Location permission is on by default, one can fix that by switching it off.
• Desktop folders has an app called DesktopFolder which can usually be installed using the AppCentre but it has not been optimized for EOS6 hence a pull request fixing the error can be found in this pull request (as of October 2021).

An Update

So I removed ElementaryOS because it was giving me the following issues -

1. Audio issues when connected without an external mic
2. Battery drain even suspended aka put to sleep
3. A little heating issue
4. Slower boot (startup) when compared to Windows
5. No audio compartmentalization, meaning audio from one app is not isolated, instead, that audio is even heard in the other app.

These issues may seem trivial but it was like a needle that one cannot find in the middle of a very soft and comfortable bed.

Hence ElementaryOS isn't an distro I would recommend for someone who has a laptop.

There's a saying - 'The devil is at his strongest while we're looking the other way.'

Like a program running in the background silently. While we're busy doing other shit. 'Daemons,' they call them. They perform action without user interaction. Monitoring, logging, notifications....

What Are Daemons?

Pronounced as day-mons. Daemons are processes that run in the background. They are usually used to perform tasks that are not directly related to the user. For example, a daemon might be used to monitor a network connection and log the traffic.

Identifying Your Daemons

A simple way of identifying daemons on your linux system is to run the pstree command.

pstree

Summoning Your Daemons

You can summon your daemons by running the service command.

service <daemon> start
service <daemon> stop
service <daemon> restart
service <daemon> status

Want to spin up a daemon?

sudo systemctl start <daemon>
sudo systemctl stop <daemon>
sudo systemctl restart <daemon>
sudo systemctl status <daemon>

Finally using systemd you can manage your daemons. A quick boot up daemon example has already been published by me here - Running Scripts On Boot.

Final Thoughts

A quick way to identify daemons even includes looking at the suffix of services. If it ends with a d it's a daemon.

Examples:

• mysqld
• systemd
• httpd
• sshd

Getting started

We will be creating a .service file in /usr/lib/systemd/system.

This will be followed by sudo nano /etc/systemd/system/discord-webhook.service. In the file you will include the following lines of code

[Unit]
Description=Discord Webhook IP sender

[Install]
WantedBy=multi-user.target

[Service]
ExecStart=python /bin/main.py
Type=simple
User=pi
Group=pi
WorkingDirectory=/bin
Restart=on-failure

Once done hit Ctrl+X followed by Y and Enter

Before actually starting the service you may want to ensure that you have the main.py code. Here is the code (you may change it to your needs)

Run this - sudo nano /bin/main.py

import socket
import requests
import urllib.request

webhookURL = '<webhook url>'
hostname = socket.gethostname()
external_ipv6 = urllib.request.urlopen('https://v6.ident.me').read().decode('utf8')
external_ipv4 = urllib.request.urlopen('https://v4.ident.me').read().decode('utf8')


data = {
    'username': 'Server IP Webhook',
    'embeds': [{
        'title': 'Server started!',
        'description': hostname + '\n' + 'IPV6' + ': ' + '`' + external_ipv6 + '`'  + '\n\n' + 'IPV4 ' + ':'  + '`' + external_ipv4 + '`',
    }]
}

result = requests.post(webhookURL, json = data)

try:
    result.raise_for_status()
except:
    pass

1. Run sudo systemctl daemon-reload
2. Run sudo systemctl start discord-webhook.service (if there is an output then your code is functioning successfully!)
3. Finally sudo systemctl enable discord-webhook.service

And you're good to go! Your script will now run at startup!

Credits - StackOverFlow

What is SCP?

SCP short for Secure Copy uses SSH (secure shell) to transfer files between 2 devices. In my experience I hae tested this out with movies as big as 3GB. Although yes it does take some time it has been one of the easiest and convenient method for me!

Syntax

scp example.mp4 pi@192.168.1.1:/home/pi/Movies

For downloading files from the other computer/server to the local machine

scp username@192.168.1.1:path_to_file path_in_the_local_machine

You have to replace example.mp4 with the file you intend to copy over, replace the pi@192.168.1.1 with your SSH address and /home/pi/Movies with the path you want your movie to be in.

Additional Points

1) If you are sending a folder don't forget to add a -r (recursive tag) 2) You can send multiple files/folders just by adding them one after another (with space) after example.mp4

Updating and Upgrading

Down to your basics, update and upgrade your system first

sudo apt update && sudo apt upgrade

InfluxDB

Installing InfluxDB

sudo apt install influxdb influxdb-client

Fiddling with Configuration

sudo nano /etc/influxdb/influxdb.conf

The values of the directories are going to be changed to tmp so that values are stored in RAM.

End result should look like this

[meta]
dir = "/tmp/influxdb/meta"

# ABRIDGED

[data]
dir = "/tmp/influxdb/data"
wal-dir = "/tmp/influxdb/wal"

Now we will bind our data to localhost only.

[http]
enabled = true
bind-address = "127.0.0.1:8086"

Changing collectd configuration

[[collectd]]
enabled = true
port = 25826
database = "collectd_db"
typesdb = "/usr/share/collectd/types.db"

Done! Now we will run the following code to enable and start InfluxDB

sudo systemctl enable influxdb
sudo systemctl start influxdb

If you now run influx in your terminal you should see it work!

Run these commands

CREATE DATABASE collectd_db
CREATE RETENTION POLICY "twentyfour_hours" ON "collectd_db" DURATION 24h REPLICATION 1 DEFAULT

"twentyfour_hours is the name of the policy, should you want to delete it later. the REPLICATION directive is only relevant for clustered systems but must be set nonetheless. Here, we only want one copy of our data.

Press CTRL + D to exit the influx prompt, and start with the next step."

collectd

Installing collectd -

sudo apt install collectd collectd-utils

Once again, open the configuration file /etc/collectd/collectd.conf and ensure the following settings are not commented out: (Usually they aren't commented out but just check)

Hostname "microserver314"

Interval 60
LoadPlugin syslog
LoadPlugin cpu
LoadPlugin cpufreq
LoadPlugin df
LoadPlugin disk
LoadPlugin entropy
LoadPlugin interface
LoadPlugin irq
LoadPlugin load
LoadPlugin memory
LoadPlugin network
LoadPlugin processes
LoadPlugin swap
LoadPlugin thermal
LoadPlugin users

What does all this mean?? The individual plugin report:

1. cpufreq: the CPU frequency
2. df: available disk space
3. entropy: available entropy for (pseudo) random number generation
4. interface: transmitted and received bytes on the network interfaces
5. irq: number of times the interrupt handler of the OS has been called
6. load: CPU load averages
7. memory: available and used main memory
8. network: write data to network servers
9. processes: number of processes and their state
10. swap: size and usage of the swap partition
11. thermal: CPU temperature
12. users: number of logged in users (via SSH, …)

Now that the plugins are loaded, we must configure some of them individually:

<Plugin df>
# This will ignore uninteresting file systems
# to keep our DB from cluttering
    FSType rootfs
    FSType sysfs
    FSType proc
    FSType devpts
    FSType tmpfs
    FSType fusectl
    FSType cgroup
    Ignore Selected true
</Plugin>

<Plugin "syslog">
# Skip messages with info label
    LogLevel "warning"
</Plugin>

Finally, we must tell collectd where to write all the data. We will set it to the host and port of InfluxDB’s collectd listener:

<Plugin "network">
    Server "127.0.0.1" "25826"
</Plugin>

Finally making it work!

sudo systemctl enable collectd
sudo systemctl start collectd

Testing The Data

influxdb

USE collectd_db
SELECT * FROM processes_value WHERE ("type_instance" = 'sleeping') ORDER BY time DESC LIMIT 1

If you get an output you are good to go!

Grafana

Finally we install the dashboard To add the Grafana APT key to your Raspberry Pi’s keychain, run the following command

curl https://packages.grafana.com/gpg.key | gpg --dearmor | sudo tee /usr/share/keyrings/grafana-archive-keyrings.gpg >/dev/null

With the key added, we can now safely add the Grafana repository to our Pi’s list of packages sources.

Use the following command on your Raspberry Pi to add the repository to the list.

echo "deb [signed-by=/usr/share/keyrings/grafana-archive-keyrings.gpg] https://packages.grafana.com/oss/deb stable main" | sudo tee /etc/apt/sources.list.d/grafana.list

Finally updating our Pi

sudo apt update

And installing it

sudo apt install grafana

Enabling and starting the Grafana service

sudo systemctl enable grafana-server
sudo systemctl start grafana-server

Now open Grafana using your host IP address and on port 3000. The password and username for the dashboard is admin.

Note - Do not like port 3000 or another service is utilizing it? Change the port in /etc/grafana/grafana.ini you have to change in /usr/share/grafana/conf/defaults.ini.

Remove the semicolon in /usr/share/grafana/conf/defaults.ini and chance the port! You're good to go!

Finally

1. Open Grafana and if not prompted, head to settings (the gear icon) and click on Data Sources
2. Add the data source as InfluxDB as given below

Done! Now just save and test! If it works you are 95% done!

Getting the beautiful output

Head over to the + sign and Import this and you shall now be seeing an output!

How is this different from any other tutorial?

The initial setup is no different from all other references but here we use Wireguard along with IPv6 to setup the server. Most of the times one cannot get a static IPv4 address to setup a PiVPN, IPv6 fixes that issue for these cases as 99% of the time it remains static. Incase your ISP changes your IPv6, well, its time to switch to a different one.

Yes some may say one can use a DDNS service but honestly its up-to you. My ISP did not allow me to port forward on my IPv4 address but IPv6 had no such restrictions. Hence this tutorial covers IPv6.

Installing the software

A small pre-requisite : Already have Pi-hole installed as you can pass all your queries through the Pi-hole.

Instead of browser plugins or other software on each computer, install Pi-hole in one place and your entire network is protected.

Once Pi-hole is setup all you have to do is run the script to install PiVPN

curl -L https://install.pivpn.io | bash

An automatic installation of PiVPN should take place and just go through with the installation GUI. There will be one part where you will be asked to input your static IPv4 address. In that just hit enter and move on, we will be making changes to that later!

After it is installed, in your terminal with your favorite editor (nano or vim) edit the Wireguard configuration file -

sudo nano /etc/pivpn/wireguard/setupVars.conf

Your end result should be something like this -

PLAT=Debian
OSCN=bullseye
USING_UFW=0
IPv4dev=wlan0
IPv6dev=wlan0
install_user=root
install_home=/home/root
VPN=wireguard
pivpnPORT=51820
pivpnDNS1=some.dns.com
pivpnDNS2=
pivpnHOST="[YOUR IPV6 ADDRESS]"
INPUT_CHAIN_EDITED=0
FORWARD_CHAIN_EDITED=0
INPUT_CHAIN_EDITEDv6=0
FORWARD_CHAIN_EDITEDv6=0
pivpnPROTO=udp
pivpnMTU=1420
pivpnDEV=wg0
pivpnNET=10.53.171.0
subnetClass=24
pivpnforceipv6route=1
pivpnforceipv6=1
pivpnenableipv6=1
pivpnNETv6="fd11:5ee:bad:c0de::"
subnetClassv6=64
ALLOWED_IPS="0.0.0.0/0, ::0/0"
UNATTUPG=1
INSTALLED_PACKAGES=(grepcidr bsdmainutils iptables-persistent wireguard-tools qrencode unattended-upgrades)

A Few Tips

• In the above options put your IPV6 (in pivpnHOST) inside the square brackets for it to work.
• The pivpnDNS1 should be automatically configured to your pi-hole address, if not, then manually configure it.
• Edit install_user and install_home to your user and home directory respectively.
• Mainly ensure pivpnforceipv6 and pivpnenableipv6 are set to 1.

Now just run

pivpn -add

to add your client to the list of devices

AND

pivpn -qr

To generate a QR code for mobile devices

Your PiVPN should be good to go! Make sure to make changes to your UFW configurations to allow UDP traffic through the inputted port (by default is 51820).

Installing Pi-hole

The easiest way of installing Pi-hole is by running the following script

curl -sSL https://install.pi-hole.net | bash

The script will do everything to install the software and convert your Raspberry Pi into a pi-hole.

Once done, you can open the web interface of the pi-hole. The web interface is simple to understand and easy to setup too.

Finally before moving on we set the DNS of our router to the IP address of our pi-hole.

Ad-lists

To add lists apart from the one used at setup we go to Group Management>Adlists

My preferred lists are -

1) Normal Hosts - https://v.firebog.net/hosts/static/w3kbl.txt

2) Adguard List - https://v.firebog.net/hosts/AdguardDNS.txt

3) EasyList - https://v.firebog.net/hosts/Easylist.txt

4) Privacy - https://v.firebog.net/hosts/Easyprivacy.txt

5) Windows Spyware - https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt

6) Anti-Malware List - https://raw.githubusercontent.com/DandelionSprout/adfilt/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareHosts.txt

7) Threat Intel by OSINT - https://osint.digitalside.it/Threat-Intel/lists/latestdomains.txt

8) Adult sites - https://blocklistproject.github.io/Lists/porn.txt

You can add all this together by adding the URLs separated by space. After doing so make sure to update the Gravity list by running pihole -g command or by heading to Tools and Update Gravity. This downloads the list and updates it.

DNS over HTTPS (DOH)

To setup DNS over HTTPS on your pi-hole we use the docs given in the official website here.

Note - Go with the manual method as the automatic did not work for me.

What will be covered here?

Although the title talks about setting up a reverse proxy we will be doing that and along with that setting up Cloudflare DNS as well as Certbot certificate generation.

Installations

Installing NGINX

Basic installation procedure just run the command

sudo apt-get install nginx

After installing NGINX just run

sudo systemctl start nginx

This should start the NGINX server at port 80. Just input your machine's IP in your browser and a welcome message should be visible.

Installing Certbot

A controversial method yes, but the only method I found convenient was installing Certbot from here using snap. Alternate methods are present too and you are free to check them out! The link above should be self explanatory.

Cloudflare DNS

To setup up a Cloudflare DNS for your domain, ensure your domain is linked to your Cloudflare account. This can be done by "verifying" you are the owner of your domain. Once that is done just head over to the DNS section as shown below.

Add a record and make it A or AAAA depending on what IP address you want to use. A is for IPV4 and AAAA is for IPV6. The IP address is your server IP address which your web servers/applications are hosted on.

If you do not want to use a DDNS service such as No-Ip you can refer this video which explains how to use Cloudflare's API and do it automatically.

NGINX Configuration

Here it is considered your certificate is ready, your DNS configured and you have NGINX installed.

A quick check should reveal an existing folder - /etc/nginx/sites-enabled/

This folder is where all your configurations for the reverse proxy will be.

Creating a configuration file

Create a file in the folder mentioned above with an extension .conf. For example test.conf.

Now all you need to do is copy paste this configuration -

server {
   listen 80;
   listen [::]:80;
   server_name test.your.domain;
   return 301 https://test.your.domain$request_uri;
 }

server {
   listen 443 ssl;
   listen [::]:443 ssl;
   server_name test.your.domain;
   ssl_certificate  /etc/letsencrypt/live/test.your.domain/fullchain.pem;
   ssl_certificate_key  /etc/letsencrypt/live/test.your.domain/privkey.pem;
   ssl_trusted_certificate /etc/letsencrypt/live/test.your.domain/chain.pem;
   ssl_prefer_server_ciphers on;

   location / {
        proxy_http_version 1.1;
        proxy_pass http://127.0.0.1:3001/;
        proxy_set_header        Host $host;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header        X-Forwarded-Proto $scheme;
        }
}

Some changes you should be making in the configuration file:

• test.your.domain should be substituted with your domain/subdomain name. The domain/subdomain you input here should be the same as inputted in the Cloudflare DNS section.
• Your ssl certificates generated by the Certbot should be in the given folder location. If not change in manually.
• If you have a website which accepts files (for example uploading of images and files) add this line - client_max_body_size 0;
• Inside location / substitute the domain in front of proxy_pass to your required domain and port.

To conclude, the same steps can be repeated as many number of times you want to create configurations for different domains/subdomains. All you have to do is create a new conf file.

💡 Tips

• There is a really good tool provided by DigitalOcean for generating configuration files: Check it out
• If you have another service for example Pi-hole running on port 80 you should switch that over to another port (lighttpd configuration: /etc/lighttpd/lighttpd.conf).

Getting Started

If you want to skip to the Github Repository here is the link.

1. Download the .uf2 file from CircuitPython for the Raspberry Pi Pico

2. Plug the device into a USB port while holding the boot button. It will show up as a removable media device named RPI-RP2.

3. Copy the downloaded .uf2 file to the root of the Pico (RPI-RP2). The device will reboot and after a second or so, it will reconnect as CIRCUITPY.

4. Download adafruit-circuitpython-bundle-7.x-mpy-YYYYMMDD.zip from here and extract it on your main device (not the Pico).

5. Navigate to lib in the recently extracted folder and copy adafruit_hid to the lib folder in your Raspberry Pi Pico.

6. Click here, press CTRL + S and save the file as code.py in the root of the Raspberry Pi Pico, overwriting the previous file.

7. Before the next step, let us enter setup mode. To enter setup mode by connecting the pin 1 (GP0) to pin 3 (GND), this will stop the pico-ducky from injecting the payload in your own machine. The easiest way to so is by using a jumper wire between those pins as seen bellow.

1. Find a script here or create your own one using Ducky Script and save it as payload.dd in the Pico.

2. Be careful, if your device isn't in setup mode, the device will reboot and after half a second, the script will run.

Enable Disable Mode

If you need the pico-ducky to not show up as a USB mass storage device for stealth, follow these instructions.

1. Enter setup mode.
2. Copy boot.py to the root of the pico-ducky.
3. Copy your payload script to the pico-ducky.
4. Disconnect the pico from your host PC.
5. Connect a jumper wire between pin 18 (GND) and pin 20 (GPIO15).
6. This will prevent the pico-ducky from showing up as a USB drive when plugged into the target computer.
7. Remove the jumper and reconnect to your PC to reprogram.
8. The default mode is USB mass storage enabled.