We know that we’re going to have some angry users as a result of this decision– but for those of you who love tags, I hope you’ll read this post and (more importantly) try out the changes we’ll be rolling out next week before you get angry or kick RescueTime to the curb.  So here’s the big news: our release next week is going to remove the concept of tagging from the user experience.  Below we’ll detail what we’re adding/changing to compensate for this and (if you’re interested) why we’re doing it.  We’ll also talk a bit of how you can get help if you are a paying customer and you are using tags.  We’ll also show you some preview screenshots.  We are really freakin’ excited about these changes–  we’re making RescueTime less work, easier to understand for new users, and (most importantly) we’re making it easier to understand and improve your own productivity.

Breakdown of the Impending Changes

Tags are going away. As we looked at how our customers were using tags (the few who were using it, much less using it successfully), we saw people tagging things like they were categories (”graphic design”, “coding”) or saw people tagging things in terms of productivity level (”work”, “personal”, “procrastination”, etc).  To allow for both of these cases, we’re expanding categories and making the act of categorizing (and creating custom categories) one billion percent easier (measure it when we launch!  It’s true!).  Further, we’re are implementing a smart default-categorization system that will provide our users with well categorized data out of the gates.  Here’s a screenshot of the new categorization UI.  It’s inline and it’s FAST.

We don’t show it in this screenshot, but there are two things we’ve added since this mockup.  First, a category dedicated to software developers (a big slice of our userbase and obviously one close to our heart!).  Two, we’ve added an inline “add custom category” selection that will make adding in your own categories close to as easy as adding in a new tag was.

So that’s attacking how most people were using tags– pretty much as categories (in fact, the average number of tags a tagged activity had in RescueTime is 1.26).

But we’re also attacking the “subjective” use case by enhancing how RescueTime scores work.  First, we’re going institute smart defaults scores for all of the top apps and sites.  This can obviously be subjective, but we think it’ll help people get up and running faster and have more meaningful data.  Second, we’re going to make scoring easier and clearer.  Here’s a screenshot of the scoring UI:

One click in any view in RescueTime and you can rescore an activity.  Note that we are exposing categorization and scoring UI in ALL reports (rather than putting them a tab deeper like in previous version).  All of these scores are now available in graph report form.  Want to see how you spend your time in terms of productivity levels?  Check out this graph:

This shows a week of my time with a breakdown of good stuff (above the line) and bad stuff) below the line.  While it’s not strictly related to this issue, this seems like a fine time to note our new URL structure, which will allow clever folk to see reports with any granularity that they want.  Want to see a graph of a day by hour?  Fine.  How about a WEEK or a MONTH by hour?  Here’s an example of the new (human readable) URL structure:

We’re hoping that all of these changes will more than compensate for the sting that some will experience from losing tags.  To learn more about why we’re doing this, read on.

Why We’re Getting Rid of Tags

We’re a metrics driven company.  Any time we release a feature, we look hard at whether people are using it and how they are using it.  When look at tags, we have some fairly ugly data to look at:

• The VAST majority of our paying customers are not using tags or not using them well (i.e. tagging Excel as “excel” doesn’t seem like a productive use of time).
• Most users have a minority of their time data tagged.
• The average number of tags a tagged activity has is 1.26… In other words, people are using tags as categories and are not taking advantage of the one-to-many powers of tags.
• When asked, most users who quit cite tagging as one of their reasons for quitting.  “Seemed like constant work with tagging”, “tagging was confusing– it was hard to make sense of my data” are very common responses.

In addition to the data, tags create some big challenges for us.  Creating visualizations of tags is difficult due to the one-to-many relationship (imagine a pie chart of tags- If Outlook is tagged as “work” and “communication”, where does Outlook time go?).  Looking at a bar graph of tags, people mistakenly assumed that if they added up the bars, that’d equal their total logged time.  Not the case!

But the biggest reason for nuking tags is that we want RescueTime (as it stands now) to be as simple as possible to make room for some of the exciting things we have in store for you.  I remember reading a comment about RescueTime that really stuck in my craw (because it was RIGHT).  Paraphrasing: “RescueTime is like a fireman walking up to you and saying, ‘Hey!  You are on fire!  You should stop being on fire!’”.  RescueTime in the coming months is going to shift into firefighting mode– and help our customers stop being on fire rather than just letting them know that they are.  We’re going to damn well live up to our name.

But why not just keep the feature and de-emphasize it? In an ideal world, this is what we’d do.  But every feature that doesn’t bring joy/satisfaction to a meaningful percentage of our users has a cost.  It clutters the UI, slows down our development process, and gives us something else to maintain until the end of time.  Too much cost, and not enough benefit, in short.

What to Do if You Have Tags

As part of this release, we’ll be assigning automatic categories and scores based on the tags people have assigned.  The 98% of our users who don’t use tags very much will have much improved data.  The 2% who use tags a lot SHOULD have improved data as well (most people who tag don’t have all of their time tagged– this will help!).  For those of you who have tags and desperately want to keep them, you’ll have the custom category capability should you need it.  If you are a paying business customer, we will help you do this from our end to minimize the pain  for you.  In other words, we’ll learn from you which tags are critical and we’ll move them into a custom category on your behalf.

For folks who are anxious about this, we apologize.  And we ask that you reserve judgement and give the adjustments a few weeks to sink in before you pass judgement.  We’re are incredibly excited about what’s coming out next week (we’ve been using it on our dev server as we’ve developed it and it’s a huge improvement on a lot of fronts).  We’re also excited about what’s on the horizon (API and a rash of cool productivity features, to name a few things)

We just quietly launched a project that we codenamed “TimePie” (based on a conversation over a year ago with Paul Buchheit, the guy who created Gmail).

One of the most common requests we get from folks is the ability to log offline time…  After all, as geeky as we might be, a good portion of our productive life is spent away from our computers.  Meetings and phone calls can be downright toxic to our productivity but, good or bad, they are important and should be measured.

Requirements

You need to be running the most recent version of the RescueTime Data Collector (http://www.rescuetime.com/download).  You also have to have a premium account of some kind (sorry, free users– we still love you, but we’ve got to charge for SOMETHING!).  You’ll need to go to your Monitoring Options page to turn on this feature (it defaults to off for all users for the time being).

How Does it Work?

First of all– lets’ get it out there– it’s not going to be about data entry.  We’re sacrificing some detail and resolution for speed and ease of use.  You literally don’t have to touch your keyboard for this feature.

After you’ve left your machine idle for more than a few minutes, RescueTime will pop up similar to the one shown below.  The clock on this timer window will keep ticking until your machine is no longer idle.

The three buttons are configurable (on the aforementioned Monitoring Options page), but will default to “Meeting” (for face to face time), “Phone Call” (for work related conference calls), and “Other Work” (for anything AFK like sketching, scribbling, whiteboarding, making dioramas, etc).  You also have the VERY large “None of your business” button.  Our goal with these default buttons is to record work time and to click the “None of your business” button for any away time that isn’t work related (lunches, breaks, etc).  However, because the buttons are configurable, you can use it however you like.

When you click on one of these buttons, it will push an activity to RescueTime in the same way it pushes information about an App or Site.  In fact, this data will exist just like an app or site and can be tagged or categorized in just the same way.

We’d love to hear your thoughts on this.  Give it a try, take it for a test drive and let us know your suggestions/thoughts in the comments.

Note: this blog post was prompted by this tweet — we’ve also gotten an email or two about the messaging change.  I’d meant to get this blog post out last week, but the release happened a few hours after I’d boarded the plan for my first vacation in a few years.  Apologies for the delay!

Last week we did a few dramatic things in terms of our business offering.  First off, our “marketing” site (the one new users see before they sign up– if you’re already a RescueTime user, you’ll have to log out to see it) is now much more business focused, with the individual offering significantly de-emphasized.  We also introduced a new product, RescueTime Pulse (employee monitoring software), which allows managers to see how employees are spending their time without the employees being able to see or control the monitoring software.  This is in contrast with our existing flagship offering (RescueTime Empower), which allows employees to see their own data and have some control over what is monitored and when.

We wanted to take a few minutes to talk about our thinking behind the new offering and what it means for RescueTime.

Our Thinking behind the Changes

• The biggest reason we’re offering the new restricted version is because people wanted it. A restricted mode offering was literally the most requested feature from our business customers.  RescueTime is a software startup, which means that our first mandate is to build something people want…  Which may or may not necessarily map to what we THINK they should want.
• Related, the site being more business-focused is a reflection of the economy in which we live.  Revenue and profit are king and we can’t expect to focus on free/consumer audiences forever.  While we will always serve that individuals, we thought the site should reflect our focus on business customers.
• The restricted offering helps us understand the value of our “in the open” offering, RescueTime Empower, which offers open and collaborative business time management software. To date, we’ve been able to show that using RescueTime in this way improves productive behavior by 9% over two months of use…  But we’ve never been able to understand how employees behave when they AREN’T using RescueTime “in the open”.  A restricted version will give us this data, and will help us understand the TRUE effect of our open offering.  9% is a pretty impressive number (annually, it can literally represents hundreds of thousands of dollars worth of productive time for even a 10-person team).  But we think we’re about to expose a much more dramatic number– and we’re excited about that!

What it means for RescueTime

• We will continue to serve new and existing individual customers.  It’s a rapidly growing audience for us (we love you guys), but free users don’t pay the bills and we don’t want to bury people in ads to make money.
• It gives us the opportunity to help business customers change how they think about employee monitoring and time management.  When you want to change the world, sometimes you have to meet it halfway and drag it the rest of the way.  Armed with real data, we can tell our customers who choose the employee monitoring route what they and their team could achieve if they embraced a more collaborative approach.
• Revenue = runway.   Again, we’re a small and young company who is trying to change how businesses and individuals think about time tracking.  That’s not going to happen overnight.   We truly believe that it IS going to happen, and this step helps insure that we’re going to be around when it does!

We still believe what we’ve always believed at RescueTime.  That time is a resource that should be tracked in the same way that any valuable resource is tracked.  That tracking time should be easy and shouldn’t interfere with being productive.  That managers and business owners should be able to see this data in aggregate to help them understand and guide their business.  That employees and individuals should be able to own their own time management, see their own time tracking data, and see how they compare to their peers.

As always, we welcome your comments (either drop a comment on this post or drop us an email at team@rescuetime.com).

Last night we switched on document and activity detail tracking for several of our most popular applications. Starting last night, if you log time in these applications and have the “Collect Window and Document Titles” setting turned on (in either your v1 RescueTime Data collector or on the Monitoring Options settings page if you have the v2 Data Collector), you’ll start to see subtotals by document or activity start to show up in your details reports.

You can expand the rows for these applications to see break down by document name, or activity (like the name of the person you were chatting with for IM applications). There’s also a new Top Activities report for applications that collect detailed information, so you can see who you chat with the most, or which spreadsheets take up the most of your time. Combined with the addition of our new keyword search filter, you’ll be able to dig deeper into your time than you’ve ever been before.

Here is a list of the applications we currently support (Windows and Mac):

Development Tools and System Utilities

Terminal
iTerm
TextMate
Xcode
Interface Builder
Aquamacs Emacs
Windows Explorer
Finder
Navicat
Command Prompt
Visual Studio

Chat / IM / Communication

MSN Messenger
Skype
Yahoo Messenger
AIM
Trillian IM
Adium IM

Office Applications

WordPad
NotePad
MS Word
MS Excel
Acrobat
PowerPoint
Visio

Other

Quicktime Player
VLC Media Player

Update: First batch of Document / Activity collecting applications is now switched on. Read about it here.

Last night we pushed out a big release for RescueTime, though it might take you a while to notice the changes given that we’ll be rolling them out slowly over the next week or two.

The three big changes to RescueTime this week are ground-up-rebuilt Data Collectors for Mac and Windows, document tracking (our most popular feature request!),  and Search filter functionality.  I’ll run through them below in a bit more detail.

New Data Collectors

The original RescueTime Data Collectors were built as prototypes.  We continued to develop and improve them as best we could, but we were never happy with the foundation of these applications.  So our scrappy Data Collector team set out to rebuild them with the goal of decreasing the memory/CPU footprint and increasing reliability and accuracy.  After months of hard work, they came through in spades.  The new Data Collectors are going out in a phased release.  We’ve been testing it with volunteers (via this post) and are now officially linking to it from our download page, which means that all new users will enjoy the new Data Collectors.  We will push out a software update to our beloved existing users once we’re confident that it performs “in the wild” as well as we’re expecting it to.  Highlights of the new Data Collectors:

• A simplified user interface – most settings will now be managed from the RescueTime.com website, and for teams and businesses using RescueTime, settings can be shared account wide, greatly simplifying installation for a large number of users
• More accurate time recording – the new RescueTime client was built from the ground up with accuracy being our key objective. Network issues, non-responsive external applications, and other issues should no longer affect the time recording functionality of RescueTime.
• More intelligent idle, sleep, and hibernation detection
• Faster updates sent to your RescueTime.com dashboard (approximately every 3 minutes!)
• A common code base between OS X and Windows – bug fixes and new features are much easier to implement
• Enhancements for enterprise wide deployments, silent installation with response files and the ability to pass configuration information at the command line
• Automated background updates – allow uninterrupted updates of the RescueTime client so you don’t lose any of your productive time
• RescueTime should now use significantly less CPU and memory than the 1.0 client version
• (Windows)RescueTime no longer depends on the Microsoft .Net 2.0 framework – which means a much simpler installation process especially in the enterprise environment
• (Windows) Support for IE8 Google Chrome 2 web browsers

Document Tracking

RescueTime has always had difficulties with vague/general applications like Microsoft Word or IM clients.  When you see that you spent 20 hours in the last month in Word, it’s sometimes challenging to understand what that means.  Were you writing a letter to your mom?  Or writing up a complex RFP document for work?  When you see that you spent 15 hours in your IM client, were you chatting with co-workers or your old college drinking buddies?

As of today (with the new Data Collectors ONLY), RescueTime supports the collection of sub-entities.  But you aren’t going to see them– yet.  Due to how Windows and Mac OS X handle windows, we can’t turn this on in a global way.  Each application requires a little bit of TLC to make it happily work with documents/sub-entities.  We’ll be phasing in visualizations of documents over the next week or two in big batches.  If there are particular apps that you’d really love to see document tracking for, please make noise in the comments or drop us a line– we’ll try to get your apps to the top of the list.  Otherwise, we obviously have a pretty good understanding of the most popular apps without or userbase and we’ll be working our way down from the top of that list.

Search Filters

We’ve added a powerful index and search capability to RescueTime which allows you to filter any report you see in RescueTime by keyword.  This is a early version of what search can do and we have several enhancements planned for the future.  The goal with search is twofold.  First, we want to allow you to get to the data that you want to see quicker– one search can often replace several click-and-wait actions.  We also feel that search is an important part of document tracking.  Once we start tracking documents, it’s going to increase the size of each persons dataset dramatically.  You can imagine how much data a manager might have at a 500-person company using RescueTime.  We want to make sure that people are able to navigate through this data easily– and we think search is a big part of this equation.

The search filter functionality we pushed last night allows to you filter any report you are viewing by keyword.  RescueTime will present a dynamically generated report of all time in that report which matches your keyword(s) in application name, document name (if supported for that app), tag, and category.

Discussed here is a mixin that extends Rails to provide an easy method to switch your database session between clean and dirty reads, otherwise known as transaction isolation level.

( jump to the code )

At Rescuetime, our most interesting analysis of tracked time requires a lot of complicated data munging on the database side. We extensively optimize the structure of the datastore and the plan of queries to produce quick results. However, there is some, however small, amount of row scanning and index hunting that is inevitable. Some of these tables are also subject to rapid, row overlapping, simultaneous insert loads.

In general, reporting or analytical access has no worries about data being up to date to the nearest microsecond, although as near real time as possible is highly desired. This near real time goal rules out an ETL type solution. Additionally, there is the cost factor. If we can make this work on one database, why build two?

In the quest for minimal stress for the online system, we introduced a method for flagging database work to be dirty reads, thus preventing any kind of locking (especially index locking) on the rows in question, and applied these where possible. This is simple enough in straight SQL, but we wanted to expose it to Rails framework in a consistent manner.

What this code does is:

1) Provide stubs in the abstract database adapter for “dirty()”, “clean()”, “reallyclean()”
2) Implement them in the MySQL adapter
3) Expose them to ActiveRecord::Base as a class method, prefixed with “isolation_”

On #1, #2: the choice of really clean versus clean simply describes the read locking strategy used. EG if you have multiple selects in same transaction on same rows, “clean” returns same result from same snapshot. However “reallyclean” will return newer rows if they exist on the later selects.

On #3, the prefix “isolation_” is added (yielding “isolation_dirty() etc.) since there is already some semantic in place for “dirty” in ActiveRecord.

For MySQL we set:

dirty = READ UNCOMMITTED
clean = REPEATABLE READ
reallyclean = READ COMMITTED

See their reference.

Just put this code in something like lib/mysql_adapter_extensions.rb in your project, then require that in some controller.

Here is the code:

With blog copy: at node.to.
Just the code: at pastie.org.

Example use:

Person.isolation_dirty()
result = Person.find_by_name params[:name] # some crazier query here
Person.isolation_clean()


RescueTime is looking for a few good volunteers to help us test out the next generation of the RescueTime installable client (version 2.0).

We expect there will be issues with the new RescueTime client and it will likely not work with all system configurations. We have done extensive testing internally and are ready to broaden our test base with a set of volunteers who don’t mind helping us out.

If you are interested in assisting in testing, head over to our GetSatisfaction forums

Features of RescueTime 2.0:

• A simplified user interface – most settings will now be managed from the RescueTime.com website, and for teams and businesses using RescueTime, settings can be shared account wide, greatly simplifying installation for a large number of users.
• More accurate time recording – the new RescueTime client was built from the ground up with accuracy being our key objective. Network issues, non-responsive external applications, and other issues should no longer affect the time recording functionality of RescueTime.
• More intelligent idle, sleep, and hibernation detection
• Faster updates sent to your RescueTime.com dashboard
• A common code base between OS X and Windows – bug fixes and new features are much easier to implement
• Enhancements for enterprise wide deployments, silent installation with response files and the ability to pass configuration information at the command line
• Automated background updates – allow uninterrupted updates of the RescueTime client so you don’t lose any of your productive time

Specific Features of RescueTime 2.0 for OS X:

• RescueTime should now use significantly less CPU and use slightly less memory than the 1.0 client version

Specific Features of RescueTime 2.0 for Windows:

• RescueTime no longer depends on the Microsoft .Net 2.0 framework – which means a much simpler installation process especially in the enterprise environment
• Support for Internet Explorer, Firefox and Google Chrome web browsers

On Monday, the entire RescueTeam dragged their collective backsides into the office a little bit late.  Weekends are sometimes challenging to recover from in terms of work schedule– but Daylight Savings Time Monday is especially painful.  Towards the end of the day we all collectively felt the day was short– and, of course, it was.  We’d reset our clocks but hadn’t quite managed to reset to the new schedule.  Which raised the question:

If we lost a chunk of our morning to Daylight Savings Time, how many other people did?  How much time was lost when people failed to reset their alarm clocks as well as their internal clocks?

It turns out that the answer is this: The average knowledge worker in our growing database spent about 16 minutes less time this past Monday than previous mondays.  Assuming there are about 36,000,000 knowledge workers in the US (the first number I could find by Googling), and assuming that they cost about $50/hr ($50,000 per year x 2 divided by about 2,000 work-hours in a year), we can say that 16 minutes is worth about $13.30 (yes, fully loaded cost is often calculated with a 2x).  Multiply that by our 36,000,000 knowledge workers and we get a cost of this convention:

$480,000,000

No idea how this affects other US workers.  And we haven’t checked if people are back to fighting trim the next day– so we suspect the cost is a bit higher.

Very soon we’re going to start releasing the RescueTime True Attention Report, detailing surprises like this one as well as a true understanding of how people really spend their time, on and off the web.  If you’d like to get the report when we release it, be sure to subscribe to this blog’s RSS feed.   If you have data that you’d love to see in a report like this, drop us a line.  Please note that the report will NEVER EVER show any individual data (anonymized or no)– just information like, “The average Outlook user spends X minutes in Outlook”.

RescueTime is a tool that helps businesses and individuals understand how they spend their time and attention– and helps them spend it more effectively.  For more information on our business offering, head to www.rescuetimeteam.com.  To check out our individual offering, head to www.rescuetime.com.

Just a note to let you know that the recent release broke the open source Linux RescueTime Data Collector.  While we don’t technically maintain or support this app, we LOVE that it exists and want to help out however we can.  From what we can tell, the Linux Collector needs a pretty simple fix, which we might eventually find time to do.  Because it’s right after a release and we’re dealing with other issues, it might be a little while.

This seems like it’s a good time to give a shout out the the folks who’ve helped out to date on the Linux client and a request for other Linux-heads to chip in.  There are a lot of good reasons to participate in open-source projects– and the RescueTime Data Collector for Linux could really use some love.

I wanted to write a quick post regarding our absolute commitment to data integrity. And what better time to do that then when we’re busy working behind the scenes to correct some data integrity problems that we discovered today.

Based on results of our validation processes after the upgrades, we found a small number of data inconsistencies for data we processed on Feb 25th and felt it was in the best interest of our users and ensuring the integrity of the data to re-run our upgrade.  No data sent has been lost and we plan on having your RescueTime data available for review as soon as possible.  Your RescueTime clients will still continue to dutifully send us accurate data, and we promise to report it back to you just as accurate  as it came in.

Thanks for your patience with this.  We’d rather fix things we broke then try to brush them under the rug and hope nobody notices.

UPDATE (3:16 AM PST THR)

We’ve verified a fix and are processing data for all paid accounts first.  Free account data will follow slightly behind.

UPDATE 2 (4:04 AM PST THR)

Paid account data for yesterday (up to Feb 25 at 8PM PST) is all processed.  We’re now processing all free account data and everything new that has come in since then.  There will be some delayed data for the next 4 hours.  It could take a few hours to see new data until we fully catch up.  The website is now available again.

UPDATE 3 (9:35 AM PST THR)

All data is current, repaired, and keeping up with live incoming client reporting. RescueTime has now returned to normal operational status.  Thanks once again for your patience.

UPDATE 4 (2:50 PM PST THR)

We’ve been tracking another issue where people’s time is coming in as something different than what it had been before (ex. MS Word is now winword) and the new app/site isn’t tagged or categorized properly.  We’re taking the site down to merge the data into the correct records.  Once that’s done everyone’s data should be correct again.  We haven’t lost any time, it’s just been put in the wrong buckets.  We’re fixing the buckets.

UPDATE (6:45 AM PST FRI)

Looks like we got everything fixed.  The website is back up.  Your data should be properly tagged and categorized once more. Many, many thanks to Montana Low, Mark Wolgemuth, and Joe Hruska for their hard work and long hours.