3: Performance Issues

This page covers performance-related issues including timeouts, rate limiting, and slow operations.

API Rate Limiting (HTTP 429)

Error: "HTTP 429 - Too Many Requests" or "Service Protection Limit exceeded"

Cause: Your requests exceed Dataverse API rate limits

Symptoms:

• Operations slow down significantly

• Sporadic failures with retry attempts

• Error messages referencing "throttling" or "rate limit"

• Progress stalls or reverses as retries are attempted

Understanding Dataverse Service Protection Limits:

Microsoft Dataverse enforces Service Protection Limits to ensure platform stability. These limits are evaluated on a 5-minute sliding window and include:

• Number of requests: 6,000 requests per user per 5-minute window

• Execution time: 20 minutes (1,200 seconds) of combined execution time per 5-minute window

• Concurrent requests: 52 concurrent requests per user

When any of these limits are exceeded, Dataverse returns an HTTP 429 error and may include a "Retry-After" header indicating how long to wait.

Solutions:

1. Switch to "Rate Safe" or "Conservative" performance preset

2. Reduce Max Concurrent Entities to 2 or lower

3. Reduce Batch Size to 100 or lower

4. Run operations during off-peak hours

5. Wait 5 minutes before retrying — the sliding window will reset

⚠ Important: If you hit the rate limit, you must wait approximately 5 minutes before retrying. Attempting to retry immediately will continue to fail and may extend the throttling period. The 5-minute sliding window means that as time passes, your oldest requests "age out" of the window, freeing up capacity.

💡 Tip: DvSchemaSync includes built-in retry logic that respects rate limits. However, if you consistently experience throttling, reduce your performance settings rather than relying on retries, as retries slow down overall throughput.

Timeout Errors

Error: "Operation timed out" or "Request timeout exceeded"

Possible Causes:

• Very large tables taking too long to sync

• Network latency issues

• Dataverse environment under heavy load

Solutions:

1. Reduce batch size to process fewer records per request

2. Sync tables individually rather than all at once

3. Run during off-peak hours when the environment is less busy

Slow Sync Performance

Symptoms: Sync operations take much longer than expected

Optimization Tips:

1. Enable Parallel Processing: Process multiple entities simultaneously

2. Increase Batch Size: Larger batches reduce API overhead (if not hitting rate limits)

3. Use Bulk Copy: Lower the Bulk Copy Threshold for large tables

4. Sync incrementally: After initial sync, only sync changed records

5. Check system resources: Ensure adequate CPU, memory, and network bandwidth

Memory Issues

Error: "Out of memory" or application becomes unresponsive

Cause: Processing very large tables or too many concurrent operations

Solutions:

1. Reduce Batch Size to use less memory per operation

2. Lower Max Concurrent Entities to reduce parallel memory usage

3. Close other applications to free up system memory

4. Switch to Conservative preset for lowest memory usage

Azure SQL Auto-Pause Timeouts

Error: Connection timeout or "Database is paused" on first sync attempt

Cause: Azure SQL Serverless databases automatically pause after a period of inactivity to reduce costs. When DvSchemaSync attempts to connect to a paused database, Azure must "warm up" the database before it can accept connections. This warm-up process can take 30-60 seconds or longer, which may exceed the default connection timeout.

Symptoms:

• First connection attempt after a period of inactivity fails with timeout

• Subsequent connection attempts succeed

• Issue recurs after database has been idle (typically 1 hour of no activity)

Solutions:

1. Retry the connection: Simply attempt the sync again after a brief wait — the database should be ready

2. Pre-warm the database: Before syncing, test the connection in Connection Management to wake the database

3. Adjust auto-pause delay: In Azure Portal, increase the auto-pause delay or disable it for frequently used databases

4. Schedule regular activity: Set up a scheduled task to query the database periodically to prevent pausing

5. Switch to Provisioned tier: If auto-pause causes frequent issues, consider switching from Serverless to a Provisioned compute tier

💡 Tip: Azure SQL Serverless is cost-effective for intermittent usage, but if you run scheduled syncs or need consistent response times, a Provisioned tier may be more suitable.

Azure SQL Tier Recommendations

Choosing the right Azure SQL tier depends on your data volume, sync frequency, and performance requirements. The following recommendations are based on typical DvSchemaSync workloads:

Serverless vs. Provisioned Compute:

⚠ Important: These are general recommendations. Your actual requirements may vary based on the number of tables synced, record counts, sync frequency, and concurrent users running reports against the database. Monitor your DTU/vCore usage in Azure Portal and scale up if you consistently exceed 80% utilization.