Automating the Newspaper and Magazine download!
By Rajan Bhattarai - 4 years and 3 months ago / Dec 2019
Originally posted on: 2018-10-03. Updated on: 2019-12-20.
This morning [October 3, 2018 GMT +545] we had this amazing discussion in dev.to twitter chat.
Time for #DevDiscuss!
— DEV Community 👩‍💻👨‍💻 (@ThePracticalDev) October 3, 2018
Tonight's topic is Personal Automation!
- When do you choose to automate a task?
- Do you have a type of task you like to automate?
- What's difficult about automating tasks? pic.twitter.com/fypNpeVcdj
Well there were many of the surprising automation stuffs and idea and here’s one i posted in reply.
I wrote a script that would download Newspaper and Entertainment Digest from the popular local media house in [ NEPAL. ] available daily, weekly. Then i setup the script with CRON Job to get them.
— //crn (@cdrrazan) October 3, 2018
I usually surf them at 8AM and tada they are already there. #devdiscuss https://t.co/4mMq0KW8eu
The Magazines and Newspaper below listed are local media and may not be suitable for your geographical region or area. So please be advised.
Behind this Idea:
The basic idea behind this automation of downloading the newspaper is I read local newspaper often as i watch news in TV rarely. Mostly i have been an internet guy so all i had is browse the epaper possible. The Media house’s online portal gives access to download the Newspaper and Magazines in PDF version as well [ which i am developing now ] so i came up with this automation idea. This is very basic scripting scenario which i wrote almost a year and half back. So, let’s get to work.
Scenario:
- 2 version of newspaper, English and Nepali published daily.
- 1 tabloid published weekly in Nepali. [ every Friday! ]
- 1 magazine published weekly in Nepali. [ every Sunday! ]
[For those who may not know, Nepali is national language spoken here in Nepal which lies in the south Asia, known as Land of Buddha and Mount. Everest.]
Working Stuff
First thing we can do is checking the system date and getting the year, month, day and week day which can be achieved in bash like this.
1
2
3
4
5
6
7
8
9
#checking the date with the system
y=$(date +%Y)
#prints the year in full format. Eg: 2017
m=$(date +%m)
#prints the month. Eg: 04
d=$(date +%d)
#prints the day. Eg: 12
w=$(date +%u)
#prints the week day. Eg: 3=wednesday
The variables used here will be useful while using with wget.
The script first checks the main directory and if the directory is not present there, it creates the folder. Then it enters into the folder. The script uses wget for downloading the paper, make sure it is installed already in the system or we could check and install if not present. But during this time, i am skipping this part.
The next step is we create a separate folder for each newspaper and magazines such that script automatically downloads epaper inside them. Skipping all the rest explanation, the major task it would do is check the folder and create if not present and download the available epaper as the script runs.
The major command we can do while downloading epaper looks like this.
1
wget -O [name-of-newspaper-or-magazine]-$y-$m-$d.pdf "http://epaper.ekantipur.com/epaper/name-of-newspaper-or-magazine/$y-$m-$d/$y-$m-$d.pdf"
Although the command block is self explaining, a little explanation.
The epaper version of the newspaper and magazines would be like this [2018-10-03.pdf] for today so we used the variable from the first step to replace in the download link for the day. The week day is extracted to check for the weekly magazines considering Monday as 1 as the tabloid is published every friday and the magazine every sunday.
The next part will be setting up a CRON JOB where we will tell the script to run every 6 or 7 AM. So we setup the cronjob like this:
crontab -e
Then insert like
0 6 * * * /home/username/script/newspaper-to-pdf.sh where the path is your script location.
Additionally we can customize in the cron to simplify your job need. But in my case, i disabled the email output. That’s all for today!
Here’s the full script.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
#!/bin/bash
#Description
#program to download the epaper from the kantipur.com
#Available Epapers by ekantipur media
# Kantipur
# The Kathmandu Post
# Saptahik
# Nepal
# 1. Clone/Download the zip.
# 2. Give execute permission to the script file by chmod +x ekantipur.sh
# 3. Run the script ./ekantipur.sh .
# Epapers will be downloaded in the following format: Newspapername-year-month-day.pdf
# Replace ~/Newspaper/kantipur/*/* in the download link with the directory to wherever you want to download!
echo "################################################################"
echo "################### News-2-PDF downloader! ####################"
echo "################################################################"
echo ""
echo "Checking Date with your system.. Please wait!"
#checking the date with the system
y=$(date +%Y)
#prints the year in full format. Eg: 2017
m=$(date +%m)
#prints the month. Eg: 04
d=$(date +%d)
#prints the day. Eg: 12
w=$(date +%u)
#prints the week day. Eg: 3=wednesday
#using wget
#downloading the kantipur[daily-nepali]
#checking the date with the system
echo "Folder may exist. It will automatically enter into the directory if present!"
cd ~
if [ -d Kantipur ]
then
echo "Directory is present. Entering into the directory..."
else
echo "Directory is not present. Creating the directory..."
mkdir Kantipur
fi
cd Kantipur
echo "Downloading Kantipur Dainik for today.. Please wait!"
if [ -d Kantipur-daily ]
then
echo "Directory is present. Entering into the directory..."
else
echo "Directory is not present. Creating the directory..."
mkdir Kantipur-daily
fi
cd Kantipur-daily
wget -O kantipur-$y-$m-$d.pdf "http://epaper.ekantipur.com/epaper/kantipur/$y-$m-$d/$y-$m-$d.pdf"
echo "Successfully downloaded Kantipur for today!"
#downloading the kathmandu post[daily-english]
cd ../
if [ -d Kathamndu-Post ]
then
echo "Directory is present. Entering into the directory..."
else
echo "Directory is not present. Creating the directory..."
mkdir Kathmandu-Post
fi
cd Kathmandu-Post
echo "Downloading The Kathmandu Post for today.. Please wait!"
wget -O thekathmandupost-$y-$m-$d.pdf "http://epaper.ekantipur.com/epaper/the-kathmandu-post/$y-$m-$d/$y-$m-$d.pdf"
echo "Successfully downloaded The Kathmandu Post for today!"
#Saptahik is released every friday!
#checking the system date to match if it is friday
#monday=1
if [ $w == 5 ]
then
echo "Downloading Saptahik for this weekend.. Please wait!"
cd ../
if [ -d Saptahik ]
then
echo "Directory is present. Entering into the directory..."
else
echo "Directory is not present. Creating the directory..."
mkdir Saptahik
fi
cd Saptahik
wget -O saptahik-$y-$m-$d.pdf "http://epaper.ekantipur.com/epaper/saptahik/$y-$m-$d/$y-$m-$d.pdf"
echo "Successfully downloaded Saptahik for this week!"
else
echo "Saptahik isn't available for today! Wait till Friday!"
fi
#Nepal magazine is released every Sunday
#checking the system date to match if it is Sunday
#monday=1
if [ $w == 7 ]
then
echo "Downloading Nepal for this weekend.. Please wait!"
cd ../
if [ -d Nepal ]
then
echo "Directory is present. Entering into the directory..."
else
echo "Directory is not present. Creating the directory..."
mkdir Nepal
fi
cd Nepal
wget -O nepal-$y-$m-$d.pdf "http://epaper.ekantipur.com/epaper/nepal/$y-$m-$d/$y-$m-$d.pdf"
echo "Successfully downloaded Nepal for this weekend!"
else
echo "Nepal magazine isn't available for today! Wait till Sunday!"
fi
echo "################################################################"
echo "################ Operation Successfully completed! ############"
