Xero - beautiful accounting software

Xero Developer Help Center

Xero Developer Community

Community > API Authentication >

Userless API Authentication

Started by Gary Greyling -   in API Authentication

With the old OAuth 1.0 flow we could generate certificates that allowed private app's to authenticate without any user-based flow. This was great for writing headless servers that synced code between our Xero org and our ERP.

Is it possible to do this with OAuth 2.0? It seems we *have* to have a user follow a flow in order to gain access tokens, making machine-to-machine integrations require some kind of UI/user flow. Please tell me there is a way to create private app authentication without having to build UI?
You'll typically need to build a small interface in order for the user to pass through the authorisation process. This should however be a one-time process as once your app has obtained the tokens it can maintain the connection without user interaction.
 

Robin B (Community Manager)  

We are in a similar situation with nearly 20 Private Apps using the old OAuth 1.0 /Certificate method to push data from our server into our clients Xero instances without any UI verification.

Surely there should be some sort of machine to machine method like service accounts that google allows?.
 

Kim Henderson  

Hey, take a look here:
https://community.xero.com/developer/discussion/109838405/

I share your frustration as we also have the server-to-server type of application where no user interaction is deemed. Will have to stay on OAuth1 until the 'client credentials' grant type spec is supported or till the end of the year and then 'dance' with indefinite token refresh hacks.

Honestly cannot really understand this decision by Xero while they clearly see how many clients may need this.
 

Andrey K.  

Server to server Privater Apps where there are no user interaction seem to have been ignored in the OAuth changeover. I'm not sure if the work around proposed in: https://community.xero.com/developer/discussion/109838405/
is going to work for us. (Is step 9 saying I need to keep refreshing my tokens in less than 30 minutes? - I thought the docs say that only the access token expires in under 30 mins and the refresh token is ok to reuse whenever to get a new access token?)
 

Kim Henderson  

Yes, that's what Xero dev team offers as a solution. You still have to auth your application for getting a refresh_token: https://developer.xero.com/documentation/oauth2/auth-flow
 

Andrey K.  

FWIW, you can use the Google OAuth playground to get a token without making your own mini-app for doing the OAuth dance:

link

However, this still leaves you with the pain point of having to refresh the token. At first I thought I could just keep the refresh token around and use it to generate new access tokens every time I want to access the API. Unfortunately, whenever you refresh the refresh token is invalidated, so you can't just keep the refresh token around forever.

Because of this, the refresh token will require some kind of central management / repository, which means some kind of locking/coordination system to ensure that different processes don't invalidate each others' tokens mid-operation.

What other APIs seem to do is make the refresh tokens immortal. Applications can request new access tokens before each operation, until the user invalidates their access, which cancels all the refresh tokens.
 

Dobes Vandermeer  

I've finally got a basic OAuth 2.0 skeleton working with the Demo site but with the way our server app works, I can't get refresh tokens to work. I always have to go the full ACCESS-TOKEN route at the start of every interaction requiring the UI user to authorise every time.

With our 20+ Xero clients we are given Advisor Access to their Xero instances so we could assist them when things go wrong and check our applications data in Xero.
In the old OAuth 1 method, this allowed us (using our Advisor access) to do all the Private App Instance/Credential setup, our end users didn't need get involved.

However with OAuth 2.0, as it's the client authorising the Application, they may have to set up the Application themselves (not us using our Advisor access) and then give us the security data so we can then do our part.
(I found OAuth 2.0 in MYOB required this)
I've spoken to someone at Xero and they couldn't tell me if this was the case and advised I need to test it with a client to see if it works!
 

Kim Henderson  

Kim:

Keep an eye out for these pitfalls:

* Refresh token can only be used if you request the offline scope
* Refresh token expires if you use it, you have to use the new refresh token and new token after that. Make sure you don't have another process somewhere refreshing the token an invalidating yours
* You cannot safely use/refresh tokens in parallel - make sure any code that accesses the API is "single-threaded". You may need some kind of central locking system or a row in your database that you use for locking
 

Dobes Vandermeer  

Thanks, some of these issues area real concern as our apps Xero interactions(exporting data to Xero) run as separate processes under their own thread (totally divorced from any UI) and there is a slight possibility that there could be overlap when the export takes a longer period of time.

I've just done some more poking around in the Xero.Net libraries looking at how it does some stuff to get a better understanding of how it does the refreshing.
 

Kim Henderson  

I noticed this morning that the Xero API (via xero-node) is accepting used refresh tokens, making this process a lot easier. Refresh tokens used to be 'one use only', and we'd frequently end up with 'invalid_grant' errors when the app got out of whack and tried to use a previously used token.
Can someone from Xero confirm that this is an intended change and will be the norm going forward?
 

Kirby Skarfe  

Take a look at this topic for the latest news on this front:

https://community.xero.com/developer/discussion/105714701/
 

Dobes Vandermeer  

Relevant quote:

Hi everyone,

The change is now live. I'll update the docs but just to clarify some of the questions from this thread:

- Refresh tokens expire after 30 days if not used. Your app will need to do a refresh at least every 30 days to keep the offline connection alive.
- If you perform a token refresh successfully you get a new refresh token with the new access token
- If, for whatever reason, you don't receive the response after performing the token refresh you can retry refreshing the old token for a grace period of 30 minutes.

We considered switching to immortal refresh tokens but we decided this was a safer compromise. Imagine, for example, a case where a developer accidentally uploads their secret and refresh token to their Github repo (it happens more than you'd think!). Refresh tokens with limited lifespans greatly reduce the likelihood that a mistake like that results in data actually being compromised.

For those who have an issue with the 30 days expiry for unused refreshed tokens - how long would you actually need?
 

Dobes Vandermeer  

I would like to echo the sentiments of those posting here. The machine-to-machine authentication mechanisms are broken in Oauth2. We need non-user authentication and authorisation as our system is running background periodic synchronisation processes to Xero and CANNOT be authorised by a user.
 

Mike O'Connell  

+1 to this issue.

We have deployment every day/ week for the app, and we only use Xero in the background process async/ S2S multiple instances (we journal transaction after payment succeed), for this concurrent process, we would need to implement persistent storage for the token, no? if not, whenever we deploy something new, we will need to give the app new tokens manually each instance.

If we have to refresh the token every time, then there's case the transaction in the middle of the time of refreshing, would be failed because of the invalidated old token? There's huge risks to us engineer to be able to report to the company if the transaction is failed, that could happen multiple times without notice.

Is there any clear timeline/ solutions from Xero to handle S2S with such cases above? We'd be happy to hear that.

Thank you.
 

Koko Gorilla  

Koko,

You can share a token between concurrent processes, you just can't refresh it in parallel. You can make a system that shares the token and uses it safely between processes.

You would need a locking system of some sort to ensure that only one process refreshes the token. Ideally it is a locking system that supports locks that can be taken in read OR read/write modes. For example you could store the tokens in a database table (encrypted) and use SELECT FOR SHARE and SELECT FOR UPDATE to take the lock.

Before each request to the Xero API you would take a "shared" / "read" lock on the current access / refresh tokens. If they are expiring soon, you take an exclusive lock on the access / refresh tokens. Then after getting the lock, check if the tokens were already refreshed while you were acquiring the lock. Then refresh the token if necessary. Now you can release the write lock on the tokens to allow other threads to use them. When you have received the response from Xero you can release the read lock on the token.

Because the lock is only held in "exclusive" mode during a refresh you will only lose concurrency every twenty minutes or so when the token is being refreshed, and probably this will take less than a second to refresh.

So, it is possible, but definitely more painful than for other APIs with immortal access credentials for server-to-server API calls.
 

Dobes Vandermeer  

> So, it is possible, but definitely more painful than for other APIs with immortal access credentials for server-to-server API calls.

Excruciatingly so. To the point of ridiculousness in my opinion. No one should have to go to the lengths you just explained just to access their own data. (you explain it very well however!)

I think the fact that Xero staff aren't chiming in on these threads says a lot. They were, but now there's so much noise over it, they've stopped. I feel like they have severely miscalculated this implementation and no longer know how to respond to these issues from the development community. The only segment of their customer base that even uses this API they've invested so much in.

It's a lemon, and they know it.
 

Aaron Cooper  

Aaron,

I feel your pain. But I doubt that their internal position is that it is a lemon. Rather, they probably have the belief that they have done the best they can with the resources they are willing to apply to this.

They may wish they could justify applying resources to additionally implement some kind of service account system for server-to-server API calls, or to have done a smoother migration between the old and new access methods, or to provide a better sandbox account for developers.

However, they have to prioritize their development efforts towards things that will improve their bottom line and people writing code to access the API are probably relatively small in number.

Most people with the resources for custom development would be investing in more up-market solutions like NetSuite, or things specifically targeted at developers like Chargify, Stripe, and so on.

Xero is meant to be more of a simple accounting app for basic business use cases, not some kind of customizable enterprise accounting software solution, and that is reflected in how they are handling the API and supporting developers (or not).


 

Dobes Vandermeer  

Dobes

You might be right. But I don't agree with the premise. Xero is a publicly traded company with cash reserves up over 400% in the 2019 FY. Of course I can't tell anyone how to run their business, but they certainly aren't a poorly resourced company developers should have sympathy for when it comes to odd implementations. At the end of the day, this is more of a decision issue than a time and resource one.

I also disagree that Xero don't want to be viewed as one of those more up-market solutions. In fact, I believe they already do. They have every intention of being an enterprise solution. And to be fair, API access is not an enterprise feature.

These use cases might be currently smaller than the wider application, I'd believe that. But it's these cases that now act as a filter for the way non-technical users see this system due to limitations communicated by technical users. I have to tell clients that Xero is either too convoluted to build upon, or completely not fit-for-purpose for all but the simplest use cases (basically - Xero is good to log into and manage accounts, but that's it).

If Xero truly only want to be a frontend simple accounting system without enterprise support, that's fine. But I don't think that's what they want themselves. Hence why I think they know they have botched this and are hesitant to respond further.
 

Aaron Cooper  

I also don't agree with the premise stated above.

For most of our clients, the interactivity between our application and Xero (via the API) is a major adoption point for both our system AND Xero. Accounting reps are basically recommending this as a coupled solution for their clients, so both us & Xero are benefitting.

We aren't trying to do anything "Enterprise", just pushing simple accounting data in/out of Xero. What has been working fine for years (for us and our clients as a server to server solution) has now been forced to implement a Client interactive OAuth 2.0 solution.

I'm fine with the need of having to change to keep things secure, but I suspect the amount of old Private Apps being used as a server to server solution is large enough to warrant a real userless, server to server solution.

 

Kim Henderson  

Just my 2 cents. I was surprised that they don't have client_credentials type grant. Implementing it would solve these machine-to-machine authentication issues forever.
 

Andrew Fionik  

I think Xero have forgot what functionality they provide to consumers of it's services. It feels they've implemented OAuth2 for security without actuallying thinking "hang on, most of our API calls come from m-2-m services and don't ACTUALLY have human interaction".

Major slip here from Xero in my opinion. There are plenty of other solutions that would have been ideal for m-2-m integration like JWT auth and IP address whitelisting for the organisation etc.

Just another bad example of a rushed implementation.
 

Marc Hull  

Looks like they have client_credentials planned "Next"
https://trello.com/c/MYURD3ft/89-custom-integrations-client-credentials
 

Rob Symonds  

It's one month away from switch off and still no machine to machine support. Whats going on?
 

Ross Hunter  

I think they consider the token refreshing approach to be sufficient and don't plan to add machine to machine support with any urgency.
 

Dobes Vandermeer  

Im told by their support that private apps get an additional 90 days, from the 31st march 2021 to switch over to oauth 2.0 from oauth 1.0. No mention from their support on when m-2-m services will be available though.
They assume that their customers need to carry the burden of the cost to modify our systems, since they have not yet implemented machine to machine authentication support..its not good
 

Ross Hunter  

I just posted to https://community.xero.com/developer/discussion/123084544/ asking where is our promised M2M service, that 90 days is ticking away...
 

Kim Henderson  

OK... so I have just hit this hurdle as well. I agree with the latest comments. This is an oversight by Xero. Completely forgot about us / overlooked a large group of users. In the short term they NEED to keep oauth 1.0 running for userless processes ( using cert ) until they have an m2m solution on oauth2. That takes the pressure off them and us. My processes run at the wee hours of the morning to generate invoices, process payments, apply receipts. User input plus the 2FA with Google Authenticator just cannot be done for unattended services. This has been working beautifully for years... don't screw it up for us please.
 

Trevor Kenney  

Hi all. I have developed a method of doing m2m using oauth 2. I've done it in php but the concepts for other languages should be the same. I'm in the process of writing up how I've done it but if anyone wants it, feel free to email me at Marc.hull@clocom.uk

Cheers
Marc
 

Marc Hull