In order to protect our servers from inadvertent service overuse or intentional attack, and to fairly distribute services to all our customers, we have implemented a rate limiting algorithm.  When fully deployed, this new algorithm will cause services to either succeed with new warnings or fail with new errors when services are consumed at too high a rate.  Programs that consume Projector services should be enhanced to handle rate limiting errors so that they can continue functioning properly.  All Service Suites will be subject to Rate Limiting:  PWS Services, Legacy Services, and Reporting Services.

This new mechanism for Rate Limiting will augment our existing throttling behavior, and will not replace it.

Best practices for programmatically handling rate limit errors is to pause all attempts to invoke Projector Services for a period of time after a rate limit error is raised, and then to retry failed invocations after that pause.  If you specify a pause that is less than the rolling window size (currently 60 seconds), you may want to iteratively use your handling until 60 seconds have elapsed since the first failure.  Please consider pausing for at least five seconds before any retries.

Algorithm Description

Projector now keeps a count of the number of “Service Hits” that are incurred during a rolling window of time per Installation (AKA Account), per User, and per Session.  Each service is allocated a “Weight” depending on its average cost to our internal processing.  When the service request is evaluated, that weight (measured in number of “hits”) is added to the previous number of service hits in each category (Installation, User, Session), and if the number exceeds a predefined limit or threshold in any of those categories, a rate limit error (or warning) is raised.  When an error is raised, the service fails before it is even internally invoked.  When a warning is raised, the service is invoked yielding a response, but that response will include the warning.

Currently, the various parameters that affect Rate Limiting are set as follows, but Projector reserves the right to change them as necessary to address the real-world usage:

• Rolling Window Size:                          Last 60 seconds
• Hit Limit per Installation/Account:      2400 / 60 seconds
• Hit Limit per User:                               1800 / 60 seconds
• Hit Limit per Session:                          1200 / 60 seconds

The actual hit count in each of the three categories and the current thresholds will be included in the error or warning returned to the consumer as part of the response when rate limiting is triggered.

PWS Service Responses

For the responses to PWS Services, when a Rate Limit Violation is encountered, the PWS Error Code that is raised is “RequestExceededRateLimit” with the Error Number of 10016.  This is true whether or not the violation triggers a warning or an error in the response.

The Error Text will contain additional information about the specifics of the violation.

In the case of a warning, the violation details will be bundled in a PWS Message of type “Warning” included in the response payload.

In the case of an error, there will be an exception triggered with a PWS Fault.  The violation details will be bundled within the PwsFault in a PwsMessage of type “Error”.

For a list of PWS Web Services and their associated weight classes/hits, please view: Rate Limiting - PWS Web Service Weights

Legacy Service Responses

For the responses to Legacy Services, when a Rate Limit Violation is encountered, the Error Code that is raised is similarly “RequestExceededRateLimit”.  This is true whether or not the violation triggers a warning or an error in the response.

The Error Message will contain additional information about the specifics of the violation.  Whether an error or warning, the “Errors” array will contain an “Error” with the details of the violation.

In the case of a warning, the service will respond with the Result of “Ok”, and the Severity of the Error will be marked “Warning”.

In the case of an error, the service will respond with the Result of “Fail”, and the Severity of the Error will be marked “Error”.

For a list of Legacy Web Services and their associated weight classes/hits, please view: Rate Limiting - Legacy Web Service Weights

When writing code that consumes Projector Services, you may want to test or simulate the situation where you are exceeding established rate limits so that you can test the code you have written to handle those limits.  A new PWS Service, PwsTestRateLimiter, is now provided for that purpose.  You can use this service to artificially increase hits within the current rolling window so that your test may include how your code responds to rate limit errors.  PwsTestRateLimiter may also be used to return current parameters in use, and the current hit counts in play.