179
Real-time data retrieval
planned
Activity
Newest
Oldest
Roderik Volman
Hi! When will this be implemented? Is this in Q1 this year? And what will be the exact solution? We actually need real, real time data retrieval. Not in one minute but instant. Thanks in advance! Team Print.com
J
Jason Little
Have you guys thought about building your own Zapier connection (or API)? I know the constraint here is Airtable's throttling of requests, but if I can control pushing some data into the Stacker base via Zapier/Integromat that would be awesome.
I use various external forms (google forms, cognito etc) and on-creation that goes into my airtable base then takes up to 5 minutes to sync to stacker, and that annoys the hell out of my users. If I could directly push that into the Stacker base, the existing process for pushing data back to Airtable via Stacker would make sure they stay in sync.
C
Cat Bradley
It would be great to get an update on this, I'm losing users because they think my app just "isn't working." here's an example of a review I received as a result, "Although you mentioned the tool was a relational database where all sections would sync up, I am finding that is not the case. In fact, I have been double entering input across sections...I continually manually refresh the tool to see if that helps, but right now, the relational database thing is not agreeable. At this juncture, there is lots of extra work on my end to try and make this a clean representation of where I am in my experience." So... yah that's a big issue.
Eriksreks
Is it possible to have access to an API that we can use to clear cache for a given record?
I could see this happening in Airtable's automations -- e.g. record gets updated --> triggers a script in Airtable to POST to api/cache/record/:recordID
S
Scott Rose
Stacker has a lot of technical problems, but this is the biggest one. There is absolutely no reason that there should be a 15-minute delay in retrieving information from Airtable. All other apps that pull data from Airtable do it INSTANTLY and in REAL-TIME. Apps like MiniExtensions, Integromat, Zapier, etc. all have REAL-TIME INSTANT DATA RETRIEVAL FROM AIRTABLE. Not Stacker. Airtable doesn’t have any restrictions on their API that would prevent Stacker from having real-time data retrieval. The only API restriction is 5 requests per second, which obviously Stacker isn’t even coming close to achieving. However you guys programmed Stacker, it seems like it was programmed incorrectly since everyone else is doing real-time instant retrieval, but you guys are not. It would also be extremely nice to have the metadata synchronize regularly, which all other platforms do just fine (such as Integromat). However, even if you don't synchronize the metadata, I am talking about just having the DATA synchronizing from Airtable to Stacker — it shouldn’t take 15 minutes, and there are no Airtable limitations that would prevent this from happening faster. All other platforms synchronize DATA from Airtable to Stacker instantly (Integromat, MiniExtensions, Zapier, etc.). There’s no reason that Stacker should be making us wait 15 minutes for updates from Airtable.
M
Mike Mauer
@Scott Rose: Don't work for Stacker, and I actually agree that this is an issue, but your examples aren't accurate here. Integromat and Zapier both do not sync instantly. They sync once every 1-15 minutes, depending on your service level (e.g. the free version offers a 15-minute sync). Similarly, it sounds like in stacker you can upgrade to a 5-minute sync time.
And sync isn't exactly the right term here. I'm assuming (I may be wrong) that it's a cron (similar to Integromat/Zapier) that is scheduled to pull from the Airtable database on regular intervals. Airtable does not have a mechanism to push data every time it changes.
Last thing I'll note is that I believe Stacker stores all data in a local database, so some changes ARE reflected in real-time. For example, if you change a field from "TRUE" to "FALSE" that displays immediately. Or if you create a new record, it shows up immediately. The only things that take 15 minutes to sync are changes that are not made directly in Stacker (e.g. If you create a new record in Airtable, not Stacker, then it'll take a while to sync).
That allllllll being said...this is a limitation and perhaps there's a way to have some elements pull from the database (e.g. list records) every time the page loads, while others (e.g. record fields/attributes) take a longer time.
Sam Davyson
@Scott Rose: Hi Scott! I understand your frustration here. We are definitely keen to get data as quickly as we can from Airtable. Unfortunately Airtable's API is limited to 100 records per request. So for Stacker to load data from Airtable in realtime if you had 1000 records in a table we'd need to call Airtable at least 10 times. This is slow and can exceed the 5 requests per second limit, particularly if your app has lots of users.
We understand how limiting the time delay can be, and we're working on various strategies to improve our performance in the future including reducing this further.
Dan Lewis
@Sam Davyson: First <3 Stacker - Go team.
Secondly: I have built custom data 'sync' functions between Airtable and Firestore and come into this pagination & rate limit issue.
I would advise whenever anyone says 'real time' I'd just read it to be 1 minute syncs. Except for update and create operations which should be optimistic updates.
My guess is you already do the latter.
The former is more complex and depends on many factors - I totally appreciate the technical challenge you are dealing with.
Heres my brain dump of understanding around this (more for the benefit of readers):
Firstly if you have access to the rate-limit status in the response from airtable then you can throttle your requests (essentially queuing them locally) - the airtable js sdk may already do this (if you are using it). If it doesn't it damn well should and you should hit them up about this.
If they don't tell you what your rate limit status is - then you have to guess, which is a bit annoying. But still doable.
Other strategies involve Airtable either giving you access to webhooks for the tables or a long-polling endpoint.
But anyway for everyone who is reading this feature request - just appreciate this is HARD. Especially when you don't control the server side of things.
<3 stacker already after just 2 hours. Go team.
ps if you want some dev support work help it me up - we are a freelancer collective down in New Zealand :)
Robert Tolmach
@Sam Davyson: Faster retrieval of new data from Google sheets would also be a huge help!
Sam Davyson
planned
We are planning to speed up the flow of data between Airtable and Stacker with several new initiatives.
We are working to increase the frequency with which we are update the cache (currently 15 mins) and also looking into making this near realtime. We'll have more to share in the next couple of weeks as we move forward with this.
J
Jim DiPiazza
@Sam Davyson: Yay! Thank You! That would be amazing!
K
Kelly McGee
@Sam Davyson: Hi Sam, any update on this or an expected timeline for release?
M
Mike Mauer
@Sam Davyson: Hey Sam - Was last weeks update (5 minute sync-time) the update to this feature request or will there be further development?
A
Ambroise Dhenain
@Thu Truong How is the implementation designed? Will you rely on webhooks or similar?
Peter Sturdivant
I tested this twice today by updating a field in Airtable's browser version and timing how long it took for the change to register in Stacker's portal (refreshing several times per minute). One test showed about 10 minutes. I gave up on the second test at minute 15, and the change had not registered in Stacker yet.
Sam Davyson
under review
J
Jason Hendrickson
Just a point of clarification - As far as I know Airtable Update doesn't refresh data, only the airtable database definition.
A
Ambroise Dhenain
@Jason Hendrickson: Airtable update may update data, that's my experience. (or it doesn't and I just got lucky)
Experienced both, once it did update data, once it did not.
C
Christelle
@Ambroise Dhenain:
To clarify the point, there's a difference between the data (records) and the airtable structure (fields, tables)
It is written in the "Airtable Update" that "If you have added, changed or removed tables or columns in Airtable, you won't see these reflected in Stacker until you run an update. This will analyze your Airtable base, and make the necessary updates in the portal."
That mean if you create new tables or columns, you need to run an update to see the changes in Stacker.
But when you go in "Setup Home", it is written: "Data is cached and will be automatically updated every 15 minutes."
I tested this morning by adding a new record in Airtable. It did show up in Stacker a while after. But I haven't run any more tests about it.
A
Ambroise Dhenain
@Christelle: Yes, that's also my understanding. My point was that it may also refresh cached data upon refresh, or may not. It's unclear if it can be related.
But I agree about the feature that sometimes it'd be useful to force a cache refresh when testing new things.
Load More
→