Govt website are bad in design, so I thought I could scrap them all at one place and they are not structured or patterned for computer scripts to scrap the available visas and its requirements automatically. So I decided to go for it. Idea was to structure the 200 Govts data and then run queries and model on it. The first part was painful to manually curate each visa from each Govt. website then use the power of open source tech to build this database and make it available to the world. I love the open source more now. I am listing here all the tech we used in the process. Now we have the first human curated and machine maintained Internet visa database for every citizen of this planet. Its like IMDb for visas!
Technology used (mostly Open Source)
The tech stack is:
Server NodeJS,
Database: ArangoDB (NoSQL Multi-model database, better than MongoDB in my opinion),
Front: VueJS SSR + Quasar Framework UI
Infrastructure: AWS EKS Kubernetes Service, CDN by Cloudflare, AWS SES. SSL Letsencrypt
Analytics: Google Analytics & data manager : System analytics by Prometheus
Code hosting & CI/CD: Gitlab AutoDevops
Code editing: Sublime & Brackets
API Management: Currently reviewing between KrakenD, Gravitee, KONG, APIMAN
Business Tech: Bitrix24 CRM, Zoho business apps, Gist Marketing Suite, Coda.io, Feedly and Pocket to keep us organized.
Creative Tech: Figma, Canva, Loom, Lumen5, Rocketium, Giphy, Nimbus snipping tool, MS Paint :)
On-Going Monitoring for visa changes:
Using above tech now we are monitoring 700+ Ministry of Tourism, Foreign affairs and immigration websites of 200 countries which alerts us on latest visa policy, law & security changes and our data team updates the database :) We are currently in public beta and improving every day.
Which are your favorite open source tech missing from the list?
You can check the database here if you like not necessary tho https://visadb.io.
Submitted May 29, 2019 at 12:06AM by dsoomro http://bit.ly/2MagZRw via TikTokTikk
No comments:
Post a Comment