We Build Shopify Apps & Extensions for Enterprise E-Commerce Companies

Shopify has strict API rate limits to prevent abuse: REST API: 2 requests/second (legacy), GraphQL: 1000 points/second (newer). Bulk operations (Bulk Metafield Apply processing 10,000 products, QR Generator fetching product data) can easily hit rate limits. Exceeding limits = throttled for 60 seconds = slow operations. Need to process thousands of products quickly without hitting limits.

GraphQL Point System:

• Each field in query costs points
• Complex queries (product + variants + metafields) = 50-100 points
• 1000 points/second = 10-20 queries/second max
• Fetching 10,000 products = 500-1000 queries = 50-100 seconds minimum

REST API Limits:

• 2 requests/second
• Fetching 10,000 products = 5,000 seconds (83 minutes!)
• Unacceptable for user experience

Rate Limit Exceeded:

• Shopify returns 429 error (Too Many Requests)
• Must wait 60 seconds before retry
• Exponential backoff required

User Experience:

• Users don't want to wait 10+ minutes for bulk operations
• Need progress indication ("Processing 5,234 of 10,000 products...")
• Can't block app UI (must run in background)

1. GraphQL Over REST:

Why GraphQL:

• More generous rate limit (1000 points/sec vs 2 req/sec)
• Fetch exactly what you need (reduce payload size)
• Bulk queries (fetch multiple products in one query)

Optimized Queries:

• Only request needed fields (reduce points per query)
• Example: Instead of fetching all product data (50 points), fetch only ID, title, tags (10 points)
• 5x more queries per second

2. Request Batching:

Fetch Multiple Products Per Query:

• GraphQL allows fetching multiple products (up to 250 per query)
• Reduces total queries: 10,000 products / 250 = 40 queries (vs 10,000 individual queries)
• Processing time: 10+ minutes → 40-60 seconds

Batch Metafield Updates:

• Shopify GraphQL supports `metafieldsSet` mutation (update multiple metafields in one call)
• Update 100 products' metafields in one mutation (vs 100 individual calls)

3. Rate Limit Bucket Algorithm:

Token Bucket:

• Start with 1000 points (bucket full)
• Each query deducts points from bucket
• Bucket refills at 1000 points/second
• If bucket empty, wait until refilled

Implementation:

• Track current bucket level (in-memory)
• Calculate query cost before sending
• If cost > bucket level, wait until bucket refills
• Ensures we never exceed rate limit (no 429 errors)

4. Exponential Backoff (Fallback):

If 429 Error Occurs:

• Retry after 1 second
• If fails again, retry after 2 seconds
• If fails again, retry after 4 seconds
• Max 5 retries, then fail gracefully

Why:

• Bucket algorithm prevents 429s 99% of time
• Backoff handles edge cases (Shopify's rate limits fluctuate)

5. Background Job Queue:

User Experience:

• User clicks "Apply Rules to 10,000 Products"
• App responds immediately: "Job queued, you'll be notified when complete"
• User can close app, job runs in background

Worker Architecture:

• Job queue (Bull + Redis)
• 5-10 worker processes (parallel processing)
• Each worker respects rate limits (shared bucket across workers)
• Progress updates stored in database (fetch via polling)

Notification:

• Email merchant when job complete
• In-app notification (next time they open app)

6. Pagination & Cursor-Based Fetching:

Shopify GraphQL Pagination:

• Cursor-based pagination (efficient for large datasets)
• Fetch 250 products per page
• Use cursor to fetch next page
• No need to fetch all pages upfront (incremental processing)

7. Caching:

Cache Product Data:

• If same product queried multiple times (e.g., rule evaluation), cache for 5 minutes
• Reduces redundant API calls
• Cache invalidation when product updated (webhook)

Cache Metafield Definitions:

• Metafield definitions rarely change
• Cache for 1 hour
• Reduces API calls by 50%+

Bulk Metafield Apply relies on webhooks for real-time automation (when product created/updated, auto-apply rules). Shopify webhooks are not guaranteed to be delivered or delivered exactly once. Webhooks can fail (network issues, app downtime), be delayed (minutes to hours), or be sent multiple times (Shopify retries). Processing same webhook twice = duplicate metafield assignments = data corruption. Need 100% reliability AND idempotency (safe to process same webhook multiple times).

Webhook Delivery Failures:

• App downtime (deployment, server crash) = missed webhooks
• Network issues = webhook lost
• Shopify doesn't guarantee delivery

Duplicate Webhooks:

• Shopify retries webhooks if no 200 OK response
• If app processes webhook but fails to respond (slow processing), Shopify sends duplicate
• Same product/create webhook received 2-3 times

Webhook Delays:

• Sometimes webhooks delayed 5-30 minutes (Shopify infrastructure issues)
• Product created at 2:00 PM, webhook received at 2:25 PM
• User sees product without metafields for 25 minutes

Out-of-Order Webhooks:

• Product updated 3 times rapidly
• Webhooks arrive in wrong order (update #3 arrives before update #2)
• Processing out-of-order = incorrect final state

Idempotency Challenge:

• Processing product/create twice = metafield assigned twice
• If rule assigns metafield value "premium", processing twice = "premium,premium" (duplicate)
• Data corruption

1.Webhook ID Deduplication:

Shopify Webhook ID:

• Every webhook has unique ID (in headers: `X-Shopify-Webhook-Id`)
• Store processed webhook IDs in database
• Before processing, check if ID already exists
• If exists → ignore (already processed)
• If new → process and store ID

Retention:

• Keep processed webhook IDs for 7 days (Shopify retries within 48 hours max)
• Auto-delete old IDs (cleanup job)

2. Immediate 200 OK Response:

Webhook Receiver Pattern:

1. Extract webhook ID and data

2. Check if already processed (idempotency)

3. Queue for async processing (don't block)

4. Respond 200 OK immediately (< 1 second)

Why:

• Shopify expects 200 OK within 5 seconds
• If timeout, Shopify retries webhook
• Async processing prevents timeout

3. Async Worker Processing:

Job Queue (Bull + Redis):

• Webhook receiver adds job to queue
• Worker processes job in background
• Worker can take 10-30 seconds (evaluate rules, update product) without timeout

Worker Logic:

1. Fetch active rules from database

2. Evaluate rules against product

3. Apply metafield assignments

4. Mark webhook as processed (idempotency)

4. Fallback Polling (for Missed Webhooks):

Daily Sync Job:

• Every 24 hours, query Shopify for recently created/updated products
• Check if rules applied (query metafields)
• If not applied → apply rules (catch missed webhooks)

Why:

• Webhooks 99% reliable, polling catches 1% failures
• Ensures eventual consistency

5. Out-of-Order Handling:

Timestamp Comparison:

• Store last processed timestamp per product
• If webhook timestamp < last processed timestamp → ignore (old webhook)
• Only process newer webhooks

Example:

• Product updated at 2:00 PM (webhook A)
• Product updated at 2:01 PM (webhook B)
• Webhook B arrives first → process, store timestamp 2:01 PM
• Webhook A arrives later → check timestamp (2:00 PM < 2:01 PM) → ignore

6. Retry Logic:

If Webhook Processing Fails:

• Worker job fails (API error, database error)
• Bull queue automatically retries (up to 3 times with exponential backoff)
• If all retries fail → move to dead-letter queue
• Alert developers to investigate

7. Monitoring & Alerts:

Webhook Health Dashboard:

• Track webhook success rate (% processed successfully)
• Track processing time (average latency)
• Track failure rate
• Alert if failure rate >5%

Dead-Letter Queue Monitoring:

• Alert if jobs in dead-letter queue
• Manual review and reprocessing

Theme Component Scheduler and Bulk Metafield Apply both use Shopify Theme App Extensions to inject dynamic content into storefront. App blocks query app API to fetch scheduled banners or metafield values. If API call is slow (>500ms), page load is delayed = poor user experience, lower conversion rate. E-commerce rule: every 100ms delay = 1% conversion loss. Need <100ms API response to avoid slowing storefront.

App Block Execution:

• Theme app block (Liquid) runs on every page load
• Makes API call to app server (fetch scheduled banners)
• If API slow (500ms-1s), page load blocked
• User sees blank space while waiting for banner

Database Query Overhead:

• Query: "SELECT active banners WHERE start_time <= NOW() AND end_time >= NOW()"
• With 1000+ scheduled campaigns, query can take 200-500ms
• Add network latency (50-100ms) = 300-600ms total

API Rate Limits (App Side):

• Storefront makes API call on every page view
• 10,000 page views/hour = 10,000 API calls/hour
• Server load spikes during traffic peaks (Black Friday)

Caching Challenges:

• Can't cache per-user (banners same for all users)
• Can cache globally BUT need to invalidate when campaign starts/ends
• Cache invalidation complex (time-based campaigns)

Page Speed Impact:

• Google PageSpeed Insights penalizes slow third-party requests
• Shopify merchants obsess over page speed (affects SEO, conversion)
• App that slows page = uninstalled

1. Aggressive Caching with Short TTL:

Redis Cache:

• Cache query result: "active banners at current time"
• TTL (Time-To-Live): 1 minute
• All page requests within 1 minute get cached result (no database query)
• After 1 minute, cache expires, query runs again

Why Short TTL:

• Campaigns can start/end at any minute
• 1-minute TTL = max 1 minute delay (acceptable)
• Balance between freshness and performance
  

2. Pre-Compute Active Banners:

Background Job (Cron):

• Every minute, query database for active banners
• Store result in Redis (before any storefront requests)
• Storefront API endpoint just reads from Redis (no database query)

Workflow:

• 2:00:00 PM: Cron job queries database, stores result in Redis
• 2:00:15 PM: Storefront request → read from Redis (instant)
• 2:00:45 PM: Storefront request → read from Redis (instant)
• 2:01:00 PM: Cron job updates Redis with new query result

Latency:

• Database query: 200ms (background, doesn't block requests)
• Redis read: <5ms (in-memory, instant)
• API response: <10ms total
  

3. CDN Caching (Shopify CDN):

Shopify CDN:

• Shopify serves storefront via CDN (global edge servers)
• Can cache app API responses at CDN edge
• Set `Cache-Control` header: `max-age=60` (1 minute)
• CDN serves cached response (no app server hit)

Benefits:

• Reduced app server load (90%+ requests served by CDN)
• Lower latency (CDN edge closer to user than app server)
• Handles traffic spikes (CDN scales automatically)

4. Inline Data in Theme (Avoid API Call):

Alternative Approach:

• Instead of app block calling API, inject data directly in Liquid
• Shopify allows app metafields on Shop object
• Store active banners in Shop metafield
• Background job updates metafield every minute
• Theme reads metafield (no API call, instant)

Tradeoff:

• Faster (no network request)
• Limited data size (metafield max 64KB)
• Suitable for small datasets (< 100 banners)

5. Lazy Loading (Non-Critical Banners):

Critical Content:

• Announcement bar (top of page) → Load immediately (critical)

Non-Critical Content:

• Sidebar banner → Load after page render (lazy)
• Footer banner → Load after page render

Implementation:

• Critical banners: Server-side render (Liquid)
• Non-critical: Client-side JavaScript fetch (after page load)
• User sees page instantly, non-critical content loads 1-2 seconds later

6. Minimize Payload Size:

API Response:

• Only return needed data (banner image URL, text, link)
• Don't return full campaign object (created_at, updated_at, etc.)
• Compress JSON (gzip)

Image Optimization:

• Serve images via Shopify CDN (auto-optimized, WebP format)
• Lazy load banner images (don't block page render)

7. Monitoring & Alerts:

API Latency Monitoring:

• Track API response time (p50, p95, p99)
• Alert if p95 >100ms
• Investigate and optimize slow queries

Cache Hit Rate:

• Track Redis cache hit rate (should be >95%)
• If low hit rate, investigate cache invalidation