Month: November 2021

Spending some quality time with the command line

Confession

It is embarrassing to admit, but I am one of those Computer Science majors not well versed in shell commands. Navigating directories, creating files, and compiling/debugging basic C programs is basically the beginning and end of my knowledge. This is what prompted to take the evening to explore a little more.

Practice, Practice

Thus, I set out to write a few simple scripts. I thought it would be neat to add a couple custom commands to my terminal. The first one prints the definition of a word to console. I accomplished this using by writing a function that accepted the word in question as an argument. The definition was requested from dict.org then that function is aliased using another function so that the argument can be passed to the function.

The second command prints out my IP address to the console. My computer is a laptop, so this is not completely frivolous. The current network configurations are found through ifconfig. The results are piped to grep which searches for lines containing “broadcast”. Then, this line is piped to awk to select the IP address portion and display it.

I noticed that by aliasing those commands that they did not persist into a new terminal window as I intended. My default shell is Z shell, so I appended the commands to the .zshrc file in the root directory. This resource script is run every time a shell is started (or restarted) so persistence was achieved. While that all sounds very basic, it was not without complications. I spent a considerable amount of time trying to figure out how to pass the argument for the dictionary word once it was aliased. I was messing around with string expansion pointlessly until I figured out the function was necessary. To my understanding, the alias arguments were being passed after the curl command which is why it only initially returned with a default.

Aspirations

This little exercise was exciting and a couple hours melted away pleasantly. It reminded me of the fun I had in Introduction to Operating Systems. Feeling reintegrated, I’ll have to think of a passion OS project for myself after this quarter. The courses from the program here at Oregon State University have offered a lot of breadth and shown me that I have only seen the tip of the CS iceberg. I enjoy that one project just becomes a stepping stone to the next project.

A small exploration to break up the semester’s work

When I first discovered API’s on the web, my world felt expanded. As my need for different kind of data increased, however, I started to feel limited by the available API’s I could find. That is how my interest was peaked by web scraping. I was familiar with the concept, but my assumption was that it would be difficult to learn and I put it off for later. I was wrong, it was much more accessible than I imagined using pre-existing Python frameworks and unlocked buckets of possibilities for acquiring new information all across the web.

Options

Before even diving into web scraping with Python. There was the question of which framework to start with. Beautiful Soup was the biggest name out of Selenium and Scrapy. Selenium has the reputation of having more set up/ learning curve, while Scrapy has more built in features such as being able to make requests and parsing html more specifically than Beautiful Soup. Ultimately, I opted to start with Scrapy, but I am interested in trying all these options at some point!

Experiment

The YouTube tutorial based project I coded up scraped images of otters from the reddit otter subreddit. Then, the pixels are mapped to ASCII characters and printed to the console. Scrapy was ridiculously easy to use. After initializing the project from the command line the set up was simply importing Scrapy into the file and specifying a start URL. The request and filtering to get the source tags with specific alt text happened all in one single command. That was all there was to it.

Overall, I was pleased with the experience, but the results need tweaking. I got some repeat images which wasn’t desirable. The ASCII art little unclear, but I have some direction for what I can change to improve.

Side Project Importance

I convinced myself that I did not have time for even a small side project. I was right, but I think I made the right call by spending time working on this. Even though web scraping is not related to current school projects, it was refreshing to try something new. I already have ideas for how I can use this technique in future projects that I have planned out and I can’t wait to see what other creative things others are doing with web scraping now that that fire is lit.