On September 1, 2023, just after 9:00 am Central Daylight Time, Simple In/Out experienced an intermittent outage lasting approximately 3 hours. During this time Microsoft Teams presence integrations were also disconnected.

Like the previous outage we suffered this Spring, this one was a confluence of events that ultimately land on us. For this, I’m sorry. While we cannot go back in time, we strive to learn from our mistakes at Simple In/Out and improve as an organization. Our customers deserve zero downtime.

Below is a technical explanation of what happened and our remedies in the interest of complete transparency.

What happened?

We’ve been rapidly shipping updates recently, setting the stage for our next new feature: Single Sign On. While we’re not ready to announce this new capability today, it’s nearing completion. In this rush, a database update caused a problem for a few customers. The database misalignment was not a big deal, but it forced us to look closely at our servers.

Unrelated, we shipped a new safety check to ensure smoothly-operating servers. This safety check noticed the database change and sent us a false negative: it reported servers broken that were running just fine. The false alarms began a chain reaction of bad reactive decisions until we discovered we were chasing non-existent problems.

What can I do?

No action is required unless using Microsoft Teams presence integration. Those users must reconnect their integrations here. Microsoft requires we contact them every 45 minutes concerning every Teams presence integration, so a persistent outage translates to Microsoft canceling all access.

What are we doing to stop this from happening again?

First, we have fixed the false positive check that caused us to react poorly.

Second, we will speed code deployments dramatically, ensuring any rollbacks take far less time. Faster code deployment will help stem further disconnects of the Microsoft Teams presence feature.

Speaking of Teams, we’re also attempting to convince Microsoft to amend their policies regarding Teams presence for more time before disconnecting. Microsoft allows anywhere from 3-30 days for all other resources, so while we’re not optimistic they will extend time regarding Teams presence we believe they can do so safely.

Thank you for reading, trusting us at Simple In/Out, and allowing us to serve you. We may have fell well short yesterday, but we’ll be better tomorrow because of our relentless pursuit in building the best in/out board on the planet.

The Simple In/Out website (www.simpleinout.com) will no longer support the Internet Explorer 11 and Microsoft Edge Legacy browsers starting September 5, 2023.

Microsoft has stopped supporting Internet Explorer 11 and Edge Legacy, so they’re no longer receiving important security updates. By supporting these old browsers, we’re slowing the adoption of modern, more secure web browsers.

Removing support for Internet Explorer 11 and Edge Legacy will speed up the development of the Simple In/Out website. We will no longer be required to spend the additional time needed to ensure that our front-end code is compatible with the old, unsupported browser engines.

Users who attempt to load the Simple In/Out website using an unsupported browser will be redirected to a landing page with instructions and links to download the latest version of Google Chrome, Safari, Firefox, or Microsoft Edge.

One of our design goals with Simple In/Out Desktop was to make it incredibly easy to update your status from anywhere. Of course, you can use our interface right from within Simple In/Out Desktop. We are particularly proud of our Taskbar menu (Windows) or Menubar item (Mac), which makes it easy to utilize Quick Picks anywhere on your computer.

Today we’ve shipped Simple In/Out Desktop 1.4.2, which adds a new Update Status shortcut you can invoke with Control-N (or Command-N on Mac). I have to say, it’s a game-changer for me and has made updating my status so much easier and quicker.

Before this new feature, users needed to click on their icon on the left-hand side to update their status with a custom comment. Now with a keyboard combination, users can update their status and return to work. No more hassle leaving what you’re currently looking at to update your status.

This new feature is a wonderful addition to Simple In/Out Desktop’s capabilities and comes directly from user feedback. If you haven’t tried it out yet, I highly recommend giving it a go and seeing how many clicks it can save you.

Simple In/Out’s integration with Microsoft Teams has become incredibly popular. Among these Teams’ features is the ability to update your Simple In/Out status when your Microsoft Teams presence changes. You can see a video on our Teams presence integration here.

Teams presence integration is game changing for users that spend all day inside Teams, but there have always been a couple of caveats. Because Microsoft doesn’t provide context about why your Teams presence changed, we must apply every presence update to Simple In/Out. While that’s typically desirable, it doesn’t work well if you also update your Simple In/Out status in other ways. An example is lunch, where you may perform a Simple In/Out status update with a great comment like “At lunch, back at 1pm”. But since you’re gone from your computer, Microsoft Teams comes along and steps on this nice comment with their presence information (like “Away”). All context is gone for all those looking at your Simple In/Out board.

Today we’ve changed the rules for how Teams presence changes are honored. Our goal is to allow Simple In/Out comments left by the user to be maintained longer. We want to encourage users to leave informative status comments and not lose them to Teams presence changes. If Microsoft tells us your presence is now “out”, we won’t update your status in Simple In/Out if you’ve already updated it to “out”. The same is true with “in” statuses.

With this feature change, users only updating their Simple In/Out status through Microsoft Teams presence will see no changes. For users that update their Simple In/Out status themselves, today’s feature change should preserve comments and allow users to utilize the Teams presence feature where they may have disabled it in the past.

Yesterday we released Simple In/Out Desktop 1.4. With this release, we’re bringing two new features to our desktop application: user management and Apple Silicon support.

User management exists in almost all of our apps (the exception being Simple In/Out TV). Now Simple In/Out administrators can add, update, archive, and delete users entirely from Simple In/Out Desktop. With these new features, we celebrate the addition of the last actions you could perform in our earlier desktop apps.

For Mac users, Simple In/Out Desktop 1.4 supports Apple Silicon. If you own an M1 or M2 processor, Simple In/Out Desktop will feel noticeably faster. As an Apple Silicon customer, I enjoy launching Simple In/Out Desktop to admire the limited number of bounces in the dock. The difference is dramatic.

We know all users will enjoy these new features and improvements.

On March 6, 2023, at approximately 9:30 am Central Standard Time, Simple In/Out experienced a 5-hour outage. Our website was largely unavailable, and our various apps experienced errors. Microsoft Teams presence integrations were also disconnected.

Yesterday was the second-longest outage in our history, rivaled only by the so-called “Netflix” Christmas Eve outage of 2012.

First, allow me to offer an apology. Unlike other outages, this one was entirely our fault. We know many customers use Simple In/Out for mission-critical tasks. Yesterday we let our customers down. We’re sorry for the disruptions during your Monday.

Below we’ll break down what happened, how we resolved the situation, what you can do, and what we intend to do to ensure this doesn’t happen again.

What happened?

As many may have guessed, this was related to the new Quick Picks update we shipped the night before. The update included a shift in how we keep a user’s status visibly current on both the web board and the Microsoft Teams’ Status tab.

A bug in this code made this operation more expensive on our system than anticipated. Another bug deployed simultaneously caused this expensive code to loop in an unanticipated place. Combined, these two bugs would cause an enormous load on our systems. We design our systems for such things, so Simple In/Out worked great for the first 17 hours.

All this changed when we deployed an unrelated bug fix Monday morning. Any time we deploy changes to Simple In/Out, we do so one server at a time. During deployments, we’re always one server short-handed. A non-issue every day but yesterday due to these bugs stressing our servers. This little one-line-of-code unrelated bug fix became the straw that broke the camel’s back.

What followed was a total system collapse. The web servers were overwhelmed with internal requests and started rejecting requests they could not complete promptly. These error responses led to our load balancers disconnecting servers due to too many errors. Next, our DNS provider realized we were throwing excessive errors at our customers and moved to protect us, which led to what many customers saw: scary DNS error pages.

How did we fix it?

We solved these issues by working with our hosting providers to identify bottlenecks (CPU) and spool up more resources. Adding more servers sounds like an easy solution, but we couldn’t definitively say more resources was the solution and wouldn’t worsen the situation. It took most of our outage time to identify the resources impacted. Once we identified the two software bugs that were the real culprits, everything made sense. Fixing those bugs restored the typical workload to our system.

What can I do?

No action is required unless using Microsoft Teams presence integration. Those users will need to reconnect their integrations here.

What are we doing to stop this from happening again?

We will establish more rigorous requirements for new features that increase our server traffic. We will explore more anomaly detection to catch traffic increases that are extraordinary or outside of our typical ebb and flow. We will also add new requirements when taking a server out of rotation during peak hours.

I hope the above deep dive into our operations provides the context behind this outage. We are firm believers in communication and transparency at Simple In/Out. Our worldwide customers trust us, and we work hard every day to maintain that trust. While we failed our customers yesterday, we’ll do our best to ensure it doesn’t happen again.

Quick Picks are a core feature, one present during Simple In/Out’s debut back in 2011. Users can, with a single click, set both their status and comment to add valuable context to their in/out board. Quick Picks also improve consistency in updates for reporting purposes.

Our Quick Picks feature has been changing over the past 16 months. We started this transition with Simple In/Out’s app for Android. Simple In/Out Desktop followed as well. Today, our new status update interface comes to simpleinout.com and the Simple In/Out integration for Microsoft Teams.

These changes begin with a new control for updating your status and typing a comment. The old radio buttons have been replaced with a control showing the in/out symbol. This control is now the default status update control, consistently everywhere you can perform a status update.

Our new Quick Picks interface separates Favorites from Recents. Previously, Simple In/Out had one list containing your favorites and recent updates. This led to confusion because users could only hide a Quick Pick from the list until they next updated their status. Favorites are now 100% curated by the user and the organization. Recents are now in a separate list and are only curated by Simple In/Out. This is the best of both worlds: a Quick Picks list entirely curated by users and a Quick Picks list automatically populated with relevant options.

In addition to this new status update interface within our Microsoft Teams integration, we’ve added your current Simple In/Out status to the Status tab. No longer will users need to head back to the Board tab to see their Simple In/Out status.

We have only one more application to update with these new interfaces: Simple In/Out for iOS. We’re hard at work and will ship this change to all iPhone users soon.

The Microsoft Teams Connector for Simple In/Out has now been restored. It was missing for the past month and a half. For a debrief on what happened, read on.

At the turn of the year, Microsoft deleted a bunch of Connectors that they claimed had “low traffic”. Among those deleted were connectors owned by a lot of big companies on the internet such as Bitbucket, Google, Mailchimp, Trello, and Zendesk. At first, no one within Microsoft developer support knew what happened. After weeks investigating they discovered their error. By then, they couldn’t get anyone else at Microsoft interested in undoing the damage they had done.

To fix this, we needed to rebuild our Connector from scratch, which we did. But then to pour salt in the wound, a glitch in the App Source process delayed our fix for another 9 days.

The silver lining is that the new Connector is better than ever. We built all configuration into the Connector itself now, which saves time and avoids an extra trip to simpleinout.com to complete the process.

We appreciate everyone hanging with us as we worked through this bug.

Microsoft Teams has long had a feature they call a Connector. Connectors allow third parties, like Simple In/Out, to send messages into the Teams chat channel of your choice. Using Connectors, our customers can relay messages to Teams when users update their statuses.

Connectors are very convenient for organizations that live and die in Teams chat. Coupled with our industry-leading Automatic Status Updates, Connectors can be a powerful tool to keep users up-to-date.

Since January 4th, 2023, Microsoft has broken connectors in Teams.

We began receiving reports from Simple In/Out users that their Microsoft Teams Connectors were no longer working. Worse yet, administrators couldn’t add a new Connector. It’s now impossible for existing customers to replace the broken Connector, and new Simple In/Out trial users cannot test Connectors at all.

We’ve been working with Microsoft since the first report. We’ve filed bug reports, held video calls, and transmitted feedback. Microsoft acknowledges the issue but have no remedy or fix at this time.

To add insult to injury, Microsoft hasn’t publicly acknowledge they have broken Connectors. With no idea of when Connectors will be working again, we have been prodding Microsoft to inform their own customers. Microsoft has avenues for notification, such as their Office Service Health dashboard or the Microsoft 365 Status Twitter account. Both show Teams issues, but this one remains conveniently absent.

When pressed on Tuesday, January 17th, we were promised Microsoft would make this public “very shortly.” After 24 hours of silence, we pressed Microsoft about their definition of “shortly” (which is clearly different from ours). Microsoft responded their “ETA seems Monday night.”

You read that right: Microsoft requires a week to publicize a bug they have been investigating for weeks. How long will a fix take if merely tweeting an advisory takes a week? We’re not optimistic.

As you can tell, I’m personally beyond-frustrated with Microsoft’s responses. We know it’s frustrating to our customers relying on Simple In/Out daily. On Microsoft’s behalf, we apologize that this feature stopped working. If you’d like to notified of Microsoft’s slow, glacial, lumbering progress fixing Connectors: reach out to us. We’ll do what they haven’t: keep you in the loop.

This week we’re proud to announce we’ve shipped a major upgrade to Simple In/Out TV. Simple In/Out TV 4 has an all-new interface and offers the most board customization we’ve ever provided in our TV app.

More Customization

Simple In/Out TV allows you to choose specific user information to display, a quantum leap over the old “our five options only” approach. You can add phone numbers, details, email address, etc. To enable these features, we also needed to allow more precision for text and profile image sizes, as well as control over column counts. We’ve also added advancing filtering by status and multiple groups. as well as multi-column sorting. You can design the TV board of your dreams with Simple In/Out TV 4.

Table View

Simple In/Out TV delivers a new table view alongside our classic card/grid view. Table views are fantastic at displaying tons of information on the screen, all in a single list that’s easy to scan with your eyes. Like our card/grid view, you control the user information displayed, the number of columns, and the sizes of text and images.

Vertical Display

Simple In/Out TV supports both portrait and landscape presentations. If you physically control your environment, you can mount televisions sideways and use Simple In/Out TV to present user lists similar to airport arrival/departure boards (this is my personal favorite way to see user lists).

Simple In/Out TV 4 is availabe today for download from either the Google Play Store or the Amazon Fire TV Store. We can’t wait to see them in the wild across the globe.