Xero has long had a "soft" limit of 1,000 transactions a month. In the past, when a business exceeded this number, Xero could become slow to respond and take a long time to generate reports.
But how strict is that 1,000-transaction limit today? Has the shift to Amazon's elastic cloud given Xero a longer life for growing businesses? Tim Richardson from Growthpath put Xero through its paces to find out.
There are questions about how well Xero performs under higher loads. After discussions with Xero and based on our own experience, we believe that businesses can interpret the limit of 1,000 invoices a month as advisory. This limit does not form part of the terms of service.
Xero has given a more concrete indication of where its actual limits lie. In January 2017, Xero published some small changes to the API limit, and these limits, which are enforced, allow a much higher load than Xero's advisory limits (5,000 API calls in a rolling 24 hour period). Hence, stress testing is relevant because you will reach the practical limits of Xero before you exceed the API limits. Exactly what are those practical limits?
Our first concern was for medium-sized wholesalers so we put together a likely scenario to see how Xero would perform. Our testing shows that Xero performs well at much higher transaction volumes than advertised. The first test results are based on three years of 2,500 transaction rows per month with matching payments uploaded via a bank CSV. The second test run was much larger.
For a medium sized wholesaler, we assume that Xero is the backend to a cloud-based inventory package such as Unleashed or Trade Gecko so we did not use Xero's inventory module.
Compared with a B2C (retail) business, a wholesaler has fewer customers and fewer invoices, but those invoices have many more lines. Most sales are on credit so reconciliation of bank feeds is important.
We carried out the initial testing in June 2015 with a standard Xero trial account and have tested the limits for clients post-AWS. One client, a games collectibles e-commerce business, sends out 5,000 invoices a month on Xero with minimal impact on performance. The tests run on a MacBook with Firefox and Chrome browsers. Data is imported with CSV files, with some browser automation for bank recs and document approvals using Python & Splinter. We also perform some uploads via the API mainly to load Xero for testing.
How does load affect Xero?
There are two patterns of use a system can see.
- Offline use, where a user is doing reports and inquiries while there is little or no live transaction entry, and
- Transactional use, when a user or API is entering sales into Xero
The back ends of modern computer systems can handle massive amounts of data very easily, so it is actually surprising to hear that cloud systems hit performance problems at fairly low levels of transactions.
The first phase of testing looks at how the size of data in a Xero file affects reporting. Because we loaded the data in a flat-file before testing began, the testing activities described below only reflect the use of the file at that time. This is like a business doing reporting after hours when there are no orders being entered, for example. Therefore, this testing looks for the impact of historical data on reporting and searching performance.
Testing showed that performance problems will come more from simultaneous load on the system rather than the volume of historical transactions. Even with transaction volumes 10 times what Xero indicates as the upper limit, Xero shows no ill-effects due to transaction history. However, the performance impact of transaction entry is greater.
Note that this is different to MYOB AccountRight desktop software, which suffers serious performance problems with high volumes of historical data. That limit is very low due to MYOB's rudimentary database. MYOB's desktop software also doesn't perform well under transaction entry. The transaction volume entered here would severely stress MYOB.
Real-Life Limits on Simultaneous Users
Xero has no limit on the number of users. However, we discovered in our testing that simultaneous entry of invoices causes problems. Xero assigns a next invoice number when a user starts entering an invoice. This number is not reserved, so if another user starts a new invoice, it will get the same number.
In fact, five users keying in invoices would all get the same number; a number is only reserved when an invoice is saved (including saving as a draft, as well as sending it straight to approval). Xero attempts to fix duplicate numbers when the invoice is saved, but this approach doesn't work very well. Our testing shows that with more than two users entering invoices at the same time, Xero overwrites invoices. This is actually the first real problem our testing showed while investigating Xero at higher loads. It's a bad bug.
We did not find a workaround for this. It is possible to manually enter invoice numbers but it would require each user to have a pool of numbers to use. Each user would need to manually keep track of the next number. A number is reserved when you save and continue editing, but this only works if you can guarantee that only one user does that at any given time.
In summary, Xero is not ready for multiple users doing order entry via the browser. Possibly it would be ok with two, but more than that is asking for trouble (i.e. lost orders).
Load Scenario 1: Medium sized wholesaler ($3m to $5m)
Period: three years
No. of sales invoices: 2,520 (70 invoices a month)
Lines per invoice: 20
No. of transactions/month: ~2,500
GST: Yes
No. of customers: 5
Bank statement: One receipt per invoice, auto-matched
This profile may not represent every "$5m wholesaler" but it is about right. Each month contains about 2500 transactions (invoices lines and bank feed). In our test environment we did not include accounts payable and payroll so it misses transaction volume in these areas. However, a business which is scaling up is likely to more likely to see transaction volume growth in the area of sales.
There are no inventory transactions in this scenario because they are managed in the supply chain front end. If you are using Xero for detailed inventory transactions our test scenario may not be accurate.
For this test, transactions were loaded via CSV (so we did not stress test the API). The files were loaded one per year (so one file of invoices, and one bank statement). Each file took only a few minutes to upload and approve the resulting draft invoices. The bank statements also loaded very quickly. The upload was designed so that Xero would immediately recognise matches (each invoice totalled to a unique amount, matching receipts on the bank statement).
However, there is no way to automatically approve the suggested matches as it is not done via the API. This was done with a script which automated the browser interface, simulating a human clicking the "OK" buttons.
Action testedResultsManual entry of a sale Quite fast. Saving the invoice took three seconds. It is about the same as an empty database.Manual entry of a journal Less than three seconds. This is a normal performanceView contact record and then sales per contact. Drill down to an invoice Fast as normal12 month P&L A few secondsGST Reconciliation Report Less than five seconds for an entire year
After this transaction load, Xero performs normally. There is no evidence of any slow-down in reporting or transaction entry.
Load scenario 2: Larger wholesaler ($10m to $15m)
Period: three years
No. of sales invoices: 7,200 (200 invoices a month)
Lines per sales invoice: 25
No. of supplier invoices: 7,200 (200 invoices a month)
Lines per supplier invoice: 5
No. of transactions/month: ~6,500
GST: Yes
No. of customers: 50
Bank statement: One receipt per invoice (for both AR and AP, auto-matched)
These tests were carried out as the only user on the system, when no processing was taking place (the 'offline' mode). Therefore these tests show the impact of the historical data loaded but not the impact of those transactions actually being entered live (the "simultaneous user" impact).
Summary: in this test database, which ended up with even more data than indicated here, offline performance was fine. Reporting, inquiries and order entry were fine, virtually indistinguishable from the smaller test set. The sheer historical volume of data is unlikely Xero.
Action testedResultsManual entry of a sale It is about the same as an empty databaseManual entry of a journal Less than three seconds. This is a normal performanceView contact record and then sales per contact. Drill down to an invoice As fast as normal!2 month P&L Less than three seconds (even with one other busy session). Normal.GST Reconciliation Report Less than five seconds for an entire year. For three years, still only a few seconds.
Evidence of a CPU bottleneck
While one session was importing invoices, an interactive session creating a new sales order faced a 45-second wait to save an invoice so performance was really suffering. This is a very intensive action.
However, while one browser session was interactively approving draft invoices, another session entering invoices experienced typical performance. New invoices were consistently saved in less than five seconds.
Using the Xero API to load invoices while entering invoices and running reports
We decided to upload invoices via the API using Python 3.5 (using the PyXero library). The Xero API is limited to 5,000 calls a day (according to documentation), but each API call makes more than one invoice.
Initially we loaded 100 invoices per call as Drafts, but this was too quick for our purposes. We wanted to do some manual interactions with Xero in the browser while the load was occurring. So we reduced it to five invoices per time: the time taken for API calls is more related to making a call than what you do with it, so it's more efficient to load 100 invoices per time than 20 calls of five each.
The invoices were the same 25-line invoices as above. Incidentally we observed that the API also performs well despite a very large volume of history in the Xero file. With decent design the API limit of 5,000 calls can actually support a very high number of transactions. Before the Jan 2017 update the limit was 1,000 a day. We considered that sufficient if used wisely so the new limit is more than enough.
While this import was running we tried reporting and invoice entry. We found the performance to be acceptable and close to normal. There were occasions were steps became slow, but we couldn't reliably repeat this.
Conclusions and workarounds
While this testing was quite rigorous it was not real world testing. Apart from the bug which makes multi-user entry of invoices at risk of data loss, the performance was much better than we expected. We stuck to wholesaler-type scenarios where transaction volume appears as invoices with quite a few lines but a relatively small number of customers (well within the 5,000 customer limit suggested by Xero). The total transaction volume was much, much higher than the limits Xero recommends and performance was good.
There are real world reports of Xero performing poorly under higher load. We would like to investigate those. We predict that these problems are likely to be in businesses with many customers and a constant stream of invoice load, perhaps via the API which is sending invoices in real time in batches of one.
The most common workaround for poor transactional performance is to buffer entry via a separate system. A retailer will usually have a point-of-sale system which records the high volume of transactions and sends only summary records to the finance system. A wholesaler may use a cloud inventory system such as Unleashed to capture the details of sales and inventory transactions.
The Australian cloud online store Neto, for example, supports (out of the box) two solutions. Firstly, it supports businesses which are using Unleashed to manage stock by sending orders there. If Neto is directly connected to Xero it offers a batching option where it consolidates invoices prior to transferring them to Xero.
If you are thinking of changing to Xero and are concerned about whether it will handle your business, please get in touch with us. We will design a test scenario matching your business and we can explore how Xero performs.
A version of this story first appeared on Growthpath.com.au
#
