How One Bug Cost Me
Over $7,000 on Google Cloud
Gusto kong i-share sa inyo ang isa sa mga pinakamalaki at pinakamasakit na pagkakamali na nagawa ko sa career ko. Ito yung kwento kung paano ako nagising isang araw na may bill na umaabot sa higit $7,000 (halos ₱417,000) mula sa Google Cloud.
Normal na araw, chineck ko yung billing para sa "Kapilya Near Me". Para akong binuhusan ng malamig na tubig. Biglang umakyat sa libo-libong dolyar ang bill sa loob lang ng isang araw. Halos lahat ng gastos, galing sa iisang service: Distance Matrix API.
Cost Breakdown
99% of the cost came from a single API.
The Culprit Code
The `applyFiltersAndRender` loop caused an N+1 issue.
Ang akala kong simpleng feature ay naging financial nightmare. Ang problema: sa bawat pag-click ng user sa filter, tumatawag ito sa getTravelTimes para sa LAHAT ng locations sa listahan. Kung may 100 na Locals, isang click lang ng user, 100 API calls agad 'yon! Ito ay umabot ng 1.4 million requests sa isang araw.
Critical Lessons
Billing Alerts
Set budget alerts. If I had an alert at $10, I would have known immediately.
API Quotas
The "kill switch." Set hard limits on requests per day. Better to error out than go bankrupt.
Batch Requests
Avoid N+1. Batch multiple destinations into a single API call whenever possible.
The Aftermath: Account Lockout
With VAT, the total hit $7,133.38. As of today, my entire Google Cloud billing account is locked.
Impact Radius
This didn't just affect Kapilya Near Me. It took down Bacolodwebdev.com, Bahanearme.com, Somedaycapsule.com, and client projects linked to the same billing account. A painful lesson in having a single point of failure.
I am currently migrating sites and negotiating with Google Support for a "goodwill credit." Stay tuned for the resolution.
Happy coding, and be a responsible dev!