Build it and they will come? Performant Search brings Flexible Reports Part 1: Key Word Filtering works!Posted: November 7, 2012
Our job was to find a long term scalable solution to the problem of Searchable Time. This post discusses our search capability and some ways to use it, now that we have reliable and speedy access to this feature. There will be a follow up post presenting the technology chosen, for those interested.
RescueTime has three features that depend on what we are calling “search”, I will be presenting two of them here: using keywords and expressions as a reporting filter with the “Search” field, and the Custom Report module (the third is “hints” in projects time entry interface).
I’ve been putting “search” in quotes (though I’ll stop that affectation now) because what we’re doing here is a bit different than a traditional Google-style search. We’re giving you a way to see a view of your RescueTime history across any span of time you choose, pivoted on your perspective of interest, eg. Categories or Activity Details or Productivity, for any activity we find that matches your search request. A “Custom Report” is just a way to save a search query for repeated use. But what does this all mean?
If you take a moment and think about it, this filtering can be very powerful. If you pick a good set of keywords, and possibly some tweaking with logical expressions (more on that later), you can get a fascinating view across your history, regardless of category, productivity, or other classification that is focused in high resolution at particular project, client, or other meme that might appear in many different applications and websites. How much time did you spend dealing with “John”? or, what is my pattern of time spent in a console versus my text editor (“terminal iterm aquamacs sublime vim”)?
Consider your document names, or folder names, email addresses, chat identities, and websites as potential members of a search expression to build these reports. The search engine will also understand logical AND and NOT and nesting. The default relationship between words is OR.
Let’s consider another example: How much did the last mini-release cost us?
You’ve got a team working on a project codenamed “Piranha”. This name appears in code filenames and directories, or Eclipse project names. It appears, with a little discipline, in your email subjects. And your support ticketing and requirements tracking system. And your marketing material’s files and web pages. And your internal chat group. And your meetings entered via offline time tracker. You get the idea– we can give a total time cost of this project, with 0 (zero) data entry across your entire organization . Well, plus any time your team spent learning about piranhas on Wikipedia (pick smart project names for best results, use logical operators to help out, eg “piranha NOT wikipedia NOT vimeo). You can then save this as a Custom Report for ongoing metrics, and side by side comparison with other ongoing custom reports.
Thank you to all our customers for sticking with us and giving feedback during the iteration of this slightly magical tool. We think search is finally fully operational.
I couldn’t resist a quick post noting The Onion’s entrée into at-the-desk analytics…
Unfortunately, since he wasn’t running RescueTime he couldn’t prove it or share it with posterity. How many of you have bested this stellar performance today?
For individual or team users, RescueTime for ChromeOS brings automated productivity management and app use statistics to ChromeBook and ChromeBox users. Get it here: RescueTime for ChromeOS to get productive today.
Google is very much on to something with ChromeOS. This system paradigm is pretty much inevitable, in some flavor, for students, schools, and the office desks. While power users may scoff at its limitations (I used to be a scoffer), the features available are constantly pushing forward– and the benefits of this totally reconsidered platform are clear.
Data always backed to the cloud. Everything encrypted. Inoculated against hardware hacking. Remotely wipeable. Centralized app asset management, user management, etc. Google really leap-frogged Microsoft on this, who should have seen this coming. It’s all those features people expect a mobile device to have, but done better, and for the desktop.
All these feature make the platform ideal for offices or schools needing a low total cost operating system, especially if just about everything the user needs to do is already on a network based service. And for those applications that aren’t, using the Chrome extension development APIs make porting access over a low cost effort.
Which brings us to RescueTime for ChromeOS. When working on our plugin update, we realized with relatively little extra work, RescueTime could deliver a full ChromeOS app for online and offline use.
Schools and businesses using the team version get the benefit of adoption and user engagement measurements– including top used apps, and most active groups in their organizations.
Just like the regular desktop app, RescueTime for ChromeOS will give the same detailed but automatic time statistics, seamlessly integrated with any other device running RescueTime, including your Android phone or tablet. Google users running RescueTime are now able to get 100% coverage of their digital life for distraction and productivity management.
Keeping track of how you are spending your time is great, but if it takes too long to track down your productivity stats, it can become a distraction on it’s own. We wanted a way to effortlessly record how we are spending our days in a way that we could easily access with a single click. So, we built RescueTime for Chrome and Firefox.
This new browser extension brings our full time-management app into your browser, and gives you a more “live” interaction with your activities as they unfold – feeding you current stats about the site you have open, as well as information about your entire day.
- Easily see how much total time you’ve spent online today
- Compare that time to your average over the past two weeks
- Understand how productive your time has been.
- See how long you’ve been on the current website (or other sites similar to the current website)
- Works with a new or existing RescueTime account
You can use the same account to track time across multiple browsers (on multiple computers, even). If you’ve got the RescueTime desktop application installed, all of your computer time will be reflected in the graphs. Just make sure you have the “I already have RescueTime on this computer” setting checked in the browser plugin.
We think it’s a great addition to the RescueTime family. We hope you’ll get as much out of it as we have.
We’ve just released a tool that lets you send your Wakoopa data export to RescueTime for processing in to your account.
The uploader is made available from the “product” section of the “account” screen of settings on our site, here is a direct link: Import time logs from other services [here]. You can upload the plain export file, or you can compress it with gzip or zip first. Here is Wakoopa’s blog post concerning their timeline: http://blog.wakoopa.com/#timeline, consult the email they sent you for instructions on getting your export. If we can get a copy of these instructions we will add the info here as well.
There are 4 steps to import:
- Make a discounted RescueTime account (2 weeks before any charge, downgrade anytime)
- Upload your import file
- Convert the data
- Import the time to your RescueTime account
If there’s any problem, contact us!
tldr; We’ve added a “Activity Details” report, that presents your normal graph views of rank, over time, and productivity of all your “detailed” activities, or documents. This view is particularly useful combined with the Search word filtering tool, which now has improved results matching.
Search Improvements: Foundation for Our New Report
Our previously announced search improvements were primarily targeted at dramatic speed gains and increased reliability. As this new infrastructure stabilized, we took a careful look at how key word filtering was working for users, and considered the great feedback our users provided in conjunction with our own analysis. Then, over the last few weeks, we’ve been tuning how we can better and more intuitively match against user’s requested search parameters– for example, if you are in Word and have a long file path for the current document you are editing, the ability to search by directory, filename, file type, application type, etc. We’ve arrived now at what seems like an effective general solution for smart indexing– but we will continue to examine the results and take feedback on how it can be further improved. All together, the much improved speed combined with the more accurate results provided us the opportunity to integrate a new report into our offering, one that makes keyword reports particularly useful for exploring how you spend your life on the computer.
A side note about terms: I use both the terms “search” and “key word filtering” due to the dual purposes of this capability. We find that predominantly users use this feature (and it’s persisted cousin, Custom Reports) for the purpose of generated reports filtered against desired match results: we call this use “key word filtering” because rather than trying to find something, you are trying to generate a filtered report with a sort of ad hoc grouping. However, users also sometimes use this feature simply to aggregate / locate time for a specific item: this is the “search” use. Finally, there is a semantic case to be made that, in general, web app users are more inclined to understand at first glance what goes in this field when it is labeled “Search”, despite that not really being its primary purpose.
A New Report: The Activity Details Report for Premium Users
A much asked for feature, our new Activity Details report provides an immediate view into the time you spend on your most urgent items, no matter what application or site you are on. If you are tracking your time in a project or for a customer, or want to understand, for example, how email time figures against your favorite design or engineering tool, this is a great resource. You can filter it with keywords to narrow down the view, and you’ll get reports that graph the top documents or pages, and a table that lets you see your app/site plus its documents. Critically, before it was impossible to see all the results of search filters in one view: you could never see matching documents/details from different apps and sites expanded together, and now you can. The old activity report is now called the Activity Summary report if you want a less noisy summary.
Navigation Changes: Integrating Search / Key Word Filtering into Regular Use
In conjunction with the above changes, we’ve rationalized how the site navigation responds to your searches and key word filtering. Again, this is an attempt to combine our own analysis with your feedback, and may be tweaked over time.
- From any report view, a new search (as in, clicking the search button) lands you on the Activity Details page. This provides you with immediate feedback for quality of your search results. Non-premium users still land on the Activities Summary page.
- Clicking search on the dashboard leaves you on the dashboard, with filtered results
- Once a search is active, it becomes sticky: if you navigate to Time reports using the side navigation, or click the “view complete report” links on dashboard widgets, your current search filter is preserved
- When viewing Activity reports, the search filter is preserved for Application or Site items linked in the table results. For example, if you search for keywords like “Seattle Atlanta”, you get a list of all apps and sites that have either of those words in their name or document details; if GMail was in the results, and you clicked the item “GMail” anywhere it occurs in the app/site column, you would get a report of all GMail items with the same keywords in its subjects and details.
- To clear out a search filter from affecting your Time Report browsing, simply click “Clear Search Filter [x]” and your screen is reloaded with the filter applied, and it is removed from all navigation points.
- Note: at this time links click *inside* the graphs themselves are not preserving the keywords, we’re continuing to explore sensible behavior for this case.
Thanks for your patience and feedback as we improve RescueTime!
We’ve been working as hard as we can to address this problem, and it has required some re-design of how our search works.
The biggest changes you should see with the new search tool are:
1) Speed. It should be fast now!
2) Search only applies to activities and documents now
3) Productivity and categories are no longer searchable in the same way as actitivies.
4) The current month is searchable already, and we are building the new index going back in time– so every day more time in the past will be added, for historical reporting.
In retrospect, while key words make sense for documents and activities, it doesn’t so much for categories or productivity scores where are known and belong to a short list. You can use a categories or productivity report from the links on the right to see those results, and actually do a search “inside” them. The ability to merge category results or multiple productivities will need to wait for new filter controls, which we are planning to introduce in the future.
Those of you have Custom Reports (which are really saved searches) may need to adjust your key words for best results. Likewise, those of you who use hints in Projects may also need to adjust your keywords.
You can think of the core issue this way: when you go to Google and you do a search, you usually care only about the top results– maybe you drill a few pages to find something. When you use “search” at RescueTime, the task is very different: you are actually doing something like “give me a report of all my time that has words like this in it”.
The key distinction here is that in the first case, you only care about retrieving a few, certainly less than 100, of the all the possible results– and if you want more, you have to make multiple requests back to the server. In our case, for your reports to be accurate with total time results, you always need every possible result to be returned. Search technology for documents is pretty well understood at this point, and there are excellent tools available for use– like Sphinx and Lucene. However, search tools are designed, for practical performance reasons among others, to operate well for the Google model of search results.
When applied to our challenge, however, we have to be much more clever about how are system is designed to allow for the “give me everything that matches” idea to work.