put up a web service that tests the use of HtmlUnit (a Java library that can be used for scraping web pages). I wanted a convenient way to test scraping from various webpages as well as test various XPaths. This app definitely helps with that. Specify the URL and XPath as parameters but they must be URL-encoded. I used this site to encode the URL parameters. For example: http://brandonjyee.com:10118/htmlunit-tester?url=www.indeed.com%2Fjobs%3Fq%3Dnode%2Bdeveloper%26l%3Dchicago%252C%2Bil&xpath=%2F%2Fdiv%5Bcontains(%40id%2C%20%27searchCount%27)%5D. The particular URL loads a query on Indeed.com for "node developer" in "Chicago IL" and the XPath selects the element that displays the result count (i.e. where id="searchCount"). Today there's about 266 job postings.
testing google maps API.
- Google Maps service example
- Geocode service example - resolves an address to a lat, long. Need lat, long to place markers in the Maps service.
deployed a background task written with Spring that writes the current date to the MySQL DB every 24h. I considered multiple options for this background process (Quartz, cron job, java timer) but the Spring @EnableScheduling with @Scheduled(fixedRate = rate) was the easiest and fastest to get working. I then just needed to register a Linux daemon that will run the jar file.
Also deployed a REST service (written with Spring Boot) that reads the data from the MySQL DB, so you can see the dates being added every 24h.
went to a Google Cloud Platform meetup. We learned how to spin up VMs and run Docker and Kubernetes. As far as site development, I'm currently trying to get a backend ingestion service running that will pull data from an online source and store it in a MySQL db. It's taking longer than expected.
got a Spring Boot REST service working. It was much easier than doing it in NodeJS, but that might be because I was already familiar with Java. Learned about Fat JARs -- java servlets that run with an embedded app server. This is the modern way of deploying web apps as opposed to creating WARs and deploying them to a standalone app server. I can see how this makes deployment MUCH cleaner and easier! It also fits in nicely with the shift from monolithic systems to a microservices architecture.
After doing a lot of googling + trial-and-error, I found out how to run the Spring app as a Linux service. See the code here
finally got the MongoPop tutorial app running on my server. See tutorial here. It still has issues with ports and DB authentication since my configuration is slightly different from the tutorial. It was relatively simple to get it running on my local computer but when trying to get it working on my remote server, there were a lot of pitfalls related to ports, DB authentication, DB user roles and read/write privileges, and running "npm start" script from PM2.
I also did some more research on MySQL vs NoSQL DBs and it seems that a NoSQL Document DB like Mongo has a very limited use case compared to a traditional RDBMS like MySQL. Most data has relationships (explicitly or implicitly) and so it makes sense to use a relational DB that can maintain the relational integrity of the data. To avoid the cost of joining multiple tables during a query, you can use a cache. This post by Sarah Mei does a great job explaining a real world case in which MongoDB was the wrong decision for the backend. She also explains that the team had accidentally conflated the use case for a cache with the use case for a persistent backend. They should be treated as two separate concepts with separate use cases.
So far, I've noticed that both Angular and React take 10+ minutes to generate a new project (or even to install someone else's project from github). I think that has to do with the node package manager both frameworks use. They both generate a huge node_modules folder (Angular ~300MB; ReactJS ~200MB with quick start kit). That's pretty annoying. The webpack dev server that's included in both is pretty darn sweet though. It refreshes instantly whenever you make changes to files and allows for very efficient local testing.
Oh, forgot to add that a common bug I'm running into is forgetting to build the apps with a target path i.e. "day5/". If I don't do that, when I upload the app to the site, I get a bunch of 404 errors b/c my app can't find the files since it has the wrong URL.
a simple html page. Mainly setting up the infrastructure on the backend. LAMP stack hosted on DigitalOcean along with a Tomcat server.
My initial goal is to do a surface dive into a few of the most popular development stacks (MEAN; MERN; LAMP; LAMJ; etc). I'll be putting up simple sites / pages in the beginning as I familiarize myself with the stacks. I want my development and deployment process to mirror (as closely as possible) those of real companies to ensure I'm learning skills that are actually useful. As I progress, I'll aim to do more involved sites with new features / improvements posted daily.