New Release - 0.9.10 - 5/31/12
Graphite 0.9.10 has been released and is now available. The packages for Whisper, Carbon, and Graphite-web are available via several sources:
- Pypi (and by extension, pip
- pypi.python.org/pypi/graphite-web/
- pypi.python.org/pypi/carbon/
- pypi.python.org/pypi/whisper/
- Github
- https://github.com/graphite-project/graphite-web/downloads
- https://github.com/graphite-project/carbon/downloads
- https://github.com/graphite-project/whisper/downloads
- Launchpad
- https://launchpad.net/graphite/0.9/0.9.10
This release contains a fabulous amount of incremental improvement over 0.9.9. Some highlights include:
- Fixes to several annoying Composer and Dashboard UI bugs
- Import of Saved Graphs into Dashboards
- Fixes to cache-full behavior for carbon-cache and carbon senders (relay and aggregator)
- Many new useful render functions and graph options
- Improvements to the rendering engine and fixes to many rendering bugs
- Support for rendering graphs as annotated SVG
- Better organized and more flexible Graphite-web config layout (local_settings.py)
Upgrading from 0.9.9 should be as simple as updating the packages. It is recommended but not necessary that local_settings.py be recreated based on the newly shipped local_settings.py.example as it includes many newly exposed settings and an improved organization and comments. Carbon's config files also have a few new settings as well to check out.
The Graphite project is also in the midst of some project changes. For those who have not yet noticed, the Graphite codebase has been moved to Github (github.com/graphite-project) and split into individual components (Graphite-web, Carbon, Whisper, and soon Ceres). The Launchpad project remains active in supporting the project with its Answers (answers.launchpad.net/graphite/) and Bugs (bugs.launchpad.net/graphite/) functionality.
Development going forward will focus on preparing what will become Graphite 0.10.0 which will include support for the Ceres database format as well as a major refactor of the Carbon daemon (nicknamed "Megacarbon"). The master branches of the project should be considered to be in an 'alpha' state for the time being and subject to backwards-incompatible changes. Fixes to the current version will be maintained in the 0.9.x project branches but no 0.9.11 version is planned for the time being.
A big thanks goes out to all those who have helped the project in contributions of time and energy in the form of code contributions, testing, discussion, and helping each other out with support questions. Additional thanks are due to Aman Gupta (tmm1) for all of his great work on the rendering engine and other fixes, Sidnei Da Silva for his work migrating the project to Github and his fixes, and everyone who's taken the time to answer questions on the Answers site and on IRC.
As always, if you need any assistance please ask a question or join us on IRC in #graphite on Freenode.
The following is a summary of changes since the last release:
New features
- Whisper
- Allocate Whisper files in chunks by default (jordansissel)
- Allow Whisper files to be allocated sparsely (jordansissel)
- Add whisper-merge command to copy data from one file to another (sidnei)
- Add whisper-dump utility (amosshapira)
- Graphite Dashboard
- New button to retrieve Graph URL (octplane)
- Add button to send email of rendered graph as attachment (bkjones)
- Allow relative ‘until’ time to be set in dashboard (daniellawrence)
- Add ability to import Graphs into dashboards from URL or Saved Graphs
- Rendering Engine:
- New minorY option to configure minor gridlines (whd)
- New alpha() function to set individual color alpha values (tmm1)
- Allow areaAlpha to set alpha values for all styles of stacked graphs (tmm1)
- New minimumAbove() function: draw only series whose min is above n (tmm1)
- New areaBetween() function: draw the area between two graph lines (tmm1)
- New holtWintersConfidenceArea() function: display area between Holt-Winters confidence bands (tmm1)
- New SVG output format with embedded graph metadata (tmm1)
- New metric whitelist/blacklist functionality using pattern files
- New filterBelowPercentile() function: remove data below n percentile from a series (tmm1)
- New removeAbovePercentile() and removeAboveValue() functions to remove outliers (tmm1)
- New removeBelowPercentile() and removeBelowValue() functions to match above counterparts
- New aliasSub() function: perform a regex search/replace on metric names (tmm1)
- New rangeOfSeries() function: reduces multiple series into the value range of each point (saysjonathan)
- New movingMedian() function: moving median, similar to movingAverage (recursify)
- New multiplySeries() function: combine series by multiplying them
- New hideYAxis option (mdeeks)
- New percentileOfSeries() function: Combines series into the value at n percentile for each point
- New tranformNull() function: transforms None values to specified (cbrinley)
- New scaleToSeconds() function: scales values based on series step (redbaron)
- New aliasByMetric() function: trims all but the last element of metric name in legend (obfuscurity)
- New uniqueLegend option to filter duplicate metric names in legend (mdeeks)
- New vtitleRight option to label 2nd Y-axis
- Carbon
- Allow flock() mode to be configured for Whisper
- Allow flushing of rrdcached before rrd data fetches (shufgy)
- Add ability to configure carbon metric prefix (jblaine)
Bug fixes
- Whisper
- Record only the last value when duplicate timestamps are sent (knyar)
- Fix rrd2whisper.py script to work with newer python-rrdtool api
- Carbon
- Fix full drain of queue after cache-full event when flow-control is enabled in both client and carbon-cache
- Fix unnecessary drop of a single metric point when cache is full
- Fix instrumentation of carbon-relay (darrellb)
- Webapp
- Fix reading of Gzip’d whisper files and remote reading of RRDs
- Fix registration of Event model in admin site
- Fix events() to work with timezone aware dates
- Fix Event model to use tagging properly and fix compatibility with MySQL (hellvinz)
- Fix compatibility of built-in json module in events and graphlot
- Fix loading of saved graphs where a target has a ‘%’ in the name
- Rendering Engine
- Fix removal of whitespace above stacked graphs with yMax setting (tmm1)
- Use powers of 2 when calculating yStep and yUnitSystem=binary (tmm1)
- Force 100% usage of vertical space when yMax=max
- Compact memcached keys to keep size under 250 after Django processing (Kevin Clark)
- Fix alignFromTrue functionality in summarize() (tmm1)
- Fix cases of mismatched units in holt-winters bootstraps (lapsu,tmm1)
- Force integer in moving average window parameter (lapsu)
- Fix incorrect cache fetch when storage dir is symlinked (mk-fraggod)
- Fix infinite loop in Y-axis render when series range is very-very small
- Fix “Undo Function” button when braces expressions are present in the target
- Fix legend column calculation (darrellb)
- Fix broken aliasByNode() (darrellb)
- Fix rendering failures when infinite values are present in series
- Fix legend text overlap with dual Y-axis mode (nleskiw)
- Fix missing hunk of graph on right side with Dual Y-axis
- Fix cactiStyle() handling of None values
- Fix rendering breakage during DST time switch
- Allow multiple named stacks of metrics (aleh)
- Fix incorrect/misaligned graphs when series with unaligned steps are mixed in a graph
- Properly shift over series that have a later start time than the graph start
- Composer
- Fix JS error on IE due to tailing list commas (reed-r-lance)
- Fix usage of + instead of %20 for spaces in URL encoding in composer view
- Fix display of a broken image rather than “No Data” when last target is removed
- Fix the loss of multiple targets when loading a saved graph with new params (vilkaspilkas)
- Fix unremovable duplicate metrics
- Dashboard
- Fix automatic edit field selection on click (octplane)
- Fix usage of browser cache-busting uniq parameter to be filtered from memcache key (sidnei)
- Fix inability to remove Graphs with duplicate target lists
Other improvements
- Carbon
- Match time units used in storage-schemas.conf with those in the webapp (ohlol)
- Only log Carbon queue fullness once (sidnei)
- Only log Carbon queue space free if it was once full (sidnei)
- Log a message with the affected filename when a Whisper update fails (bmhatfield)
- Move carbon instance logs to their own own directory to prevent clobbering
- Prevent carbon-aggregator from clobbering aggregated values when aggregating to same-name
- Add SSL option to amqp publisher (sidnei)
- Remove duplicate dot metric path filtering for performance (drawks)
- Refactor of schema validation to give more informative errors
- Add reloading of rewrite-rules and aggregation-schemas for consistency
- Webapp
- Refactor settings.py to allow more complete configuration in local_settings.py
- Make Graphite compatible with Django 1.4
- Add jsonp support for /browser endpoint
- Make it harder to break metric browsing with a bad DATA_DIRS entry
- Rendering Engine:
- Make asPercent() much more flexible and useful
- stddev() function made more robust
- Allow metrics to begin with a braces-wildcard
- Prevent drawAsInfinite() lines from affecting Y axis height (bmhatfield)
- Pass through time with secondly rather than minutely resolution to rrdfetch (tmm1)
- Tree branches should display above all leaves (mdeeks)
- Add alignToInterval to hitcount() function similar to summarize() (jwoschitz)
- Fix PieGraph missing function
- Allow timeShift() to shift forward as well as backward
- Composer
- Don’t reorder targets when applying functions
- Refactor of Graph Options menu
- Dashboard
- Explicitly size img tags to keep scroll position intact during reloads
- Default the navBar as collapsed when loading an existing dashboard view
- Show wildcards in top nav browsing view
- Allow dashboards to have any character in title (octplane)
- Make “Remove All Graphs” and “Change Size” dialogs modal (dannyla)
- Make the new “north” navbar the default
New Release - 0.9.9 - 10/6/11
Graphite 0.9.9 is now out and available for download. It available through PyPI (pypi.python.org/pypi) and the Launchpad project page (https://launchpad.net/graphite).
This is a very substantial release. To give you an idea, the 0.9.8 release was cut from trunk around revision 380 while 0.9.9 was cut from revision 589, so that's almost as many commits as Graphite has ever had just since 0.9.8. The full changelog is too big for me to assemble nicely unfortunately, but I will try to cover all the important bits and if you're really curious you can see all the changes at bazaar.launchpad.net/~graphite-dev/graphite/main/changes
There are some really important things you need to know if you're upgrading from an earlier release (even trunk). Read all the change summaries below please!
API Changes
- There have been API changes in whisper, carbon, and the webapp. If you are upgrading to 0.9.9 YOU MUST UPGRADE ALL 3 PACKAGES, if you mix 0.9.8 whisper with 0.9.9 carbon for example, it won't work. Upgrade all 3, and don't forget to use the —force.
- The webapp has a new dependency on django.tagging (you should be able to simply 'pip install django-tagging')
New Default Behavior
- We've addressed a security vulnerability with receiving pickled datapoints, see Bug #817247. This affects you in that the new default behavior is to use a more secure unpickler, which is slightly slower than the standard insecure unpickler. To revert to the less secure but faster approach previously used, you have to set USE_INSECURE_UNPICKLER=True in your carbon.conf.
Revamped Dashboard UI
- You can now use all the composer functionality by clicking on a dashboard graph
- You can drag and drop to move graphs around (and hover-drop to combine them!)
- There is an awesome new auto-completer interface available by going to the Dashboard menu, Configure UI, Completer. This may become the default in the future because its so awesome. (pro tip: try using the dashboard completer with \* instead of * for some really powerful 'group by' functionality)
Other Stuff
- Tons of readthedocs.org improvements, also the example config files now have some great comment documentation
- Whisper now supports rollup aggregation methods other than averaging. The default is still to average but there a new aggregation-schemas.conf (see Bug #853955)
- To learn about the new metric metadata API that can be used to configure custom rollup aggregation methods read my answer to https://answers.launchpad.net/graphite/+question/173304 (you can skip the question part if you just care about the new API)
As for the current development focus, I can now finally work on the long-awaited merge of the 1.1 branch into trunk. The Ceres database will be in the next release, I'm going to try and merge it in (including the new refactored storage API) in the next week or so. I'll announce on graphite-dev when its available for testing. My aim is to get it fully documented for 1.0, which I'm targetting for end of this year. There might be an 0.9.10 first, depending on how many bugs are found in 0.9.9.
As always, thanks to everyone who has contributed, especially the following rockstar crew that made some major contributions in the past few months:
- Aman Gupta (tmm1)
- Nick Leskiw (nleskiw)
- Sidnei da Silva
New Book - The Architecture Of Open Source Applications - 5/25/11
Hey everyone, a cool new book (www.aosabook.org/) is out that discusses the architecture of many open source software projects. Each chapter is written by a core contributor (typically the original author) of each project and explains the design decisions that were made and why. There happens to be a chapter on Graphite written by yours truly, chrismd. This is a not-for-profit book and all proceeds go to Amnesty International. If you want to buy the book, I'd recommend buying it straight from the publisher, Lulu.com (www.lulu.com/browse/search.php?fListingClass=0&fSearch=architecture+of+open+source+applications) as that maximizes the amount that gets donated. Plus Lulu is just plain awesome.
In other news, I recently left Sears and started working at Google in Mountain View. I'm enjoying it very much it but there has definitely been an impact on my recent contributions (or lack thereof) to Graphite. I just wanted to say that I am still very much planning on spending time continuing to work on Graphite. I imagine I won't get much done in the next month or so, but sooner than later I aim to finish up the new documentation for readthedocs.org. That is the next goal, no other features/projects until the docs are in tip-top shape. In the mean time, I really appreciate all of the community contributions on the Launchpad forums and IRC channel, I am still way behind on answering questions so any and all help is appreciated.
New Release - 0.9.8 - 4/3/11
Graphite 0.9.8 is now out and available for download. It available through PyPI (pypi.python.org/pypi) and the Launchpad project page (https://launchpad.net/graphite).
This release is a major step forward for Graphite, with a long list of substantive enhancements only 3 months after the last release. One of the highlights is the move of our documentation to readthedocs.org, the docs are now built using Sphinx and they live in trunk under the 'docs' folder. Just commit any changes and readthedocs.org will automatically update by pulling changes from launchpad nightly.
A special thanks goes out to AppNexus (appnexus.com/), who sponsored the development of two awesome new features. First is the new carbon-aggregator daemon. This new daemon lets you configure the calculation of aggregate metrics at storage time instead of using a heavy-weight sumSeries or averageSeries at rendering time. This daemon can also rewrite metric names. You manage it like the other two carbon daemons, via carbon.conf. Documentation on configuring carbon-aggregator will be coming soon.
AppNexus also sponsored the development of the new Dashboard UI. This new interface allows you to put together dashboards containing many graphs quickly and easily. You can save a dashboard and view it later. Note that this is a basic implementation for now,
Beyond that, there are many other new features so please read through the changelog carefully.
Changes
- New carbon-aggregator daemon can compute your aggregate metrics
- New Dashboard UI
- Upgraded to ExtJS 3.3
- All Documentation is moving to Sphinx in our bzr branch, HTML builds of it are hosted by readthedocs.org (graphite.readthedocs.org/)
- The recommended Apache setup is now officially mod_wsgi and not mod_python.
- New metric pattern syntax, eg. example.{foo,bar}.metric, matches both example.foo.metric and example.bar.metric
- Y-axis now draws much more useful labels for values much less 1
- The YAxis=left|right parameter has been renamed to yAxisSide=left|right
- Rewrote webapp/render/grammar.py to be much more readable
- Added new json api call /metrics/expand/?query=foo.* -> \["foo.bar", "foo.baz", …\]
- Added debugging manhole in carbon-cache.py (ssh-accessible python intepreter interface into carbon at runtime)
- Added new hitcount function (thanks to Shane Hathaway)
- The "User Graphs" tree now works properly for usernames that contain dots
- Fixed data roll-up bug in whisper
- Added AUTOFLUSH option in whisper/carbon for synchronous I/O
- and as always, many more smaller bug fixes
New Release - 0.9.7 - 1/8/11
UPDATE - if you downloaded the original 0.9.7 package of the graphite webapp and had problems, please try the updated 0.9.7c package!
A little late but better than never, Graphite 0.9.7 is now out and available for download. It available through PyPI (pypi.python.org/pypi) and the Launchpad project page (https://launchpad.net/graphite). Here is a quick-rundown of the new features and some nice bug fixes:
Features
- Composer UI menus have been updated to reflect all currently available functions and options
- New threshold() function allows you to draw a horizontal line with a custom color and legend name (though color and legend name are not available through composer UI yet)
- New summarize() function allows you to draw data at a lower precision than it is stored at (ie. draw hourly datapoints for minutely data)
- New group() function allows you to specify a collection of metrics for passing to other functions that require a single arg, without using wildcards.
- Retention configurations support a new more convenient syntax (see Bug #697896)
- Carbon's logging of every whisper update can be disabled now (set LOG_UPDATES = False in carbon.conf)
- Carbon-relay can now specify ports for remote carbon-caches
- Timezones can now be specified at render-time using Olson timezone names (see pytz)
- Saved MyGraphs now support a hierarchical structure when dots are used in the saved graph names
- By popular request, carbon now ignores improperly formatted datapoint lines rather than disconnecting the client (Bug #589476)
- X-axis labeling has been revamped to avoid overlapping and confusing labels
- RPM and source RPM packages are available for download. Note that they currently do not check dependencies and do not perform post-install tasks. This means they are suitable for upgrades but the usual install doc will need to be followed for new installations. Please contribute feedback regarding these packages so we can make them work out of the box on Fedora and CentOS.
Bugs Fixed (woefully incomplete)
- Bug #528228 - fixed 'tz' parameter for specifying custom timezone at render-time
- Bug #676395 - fixed timeShift() function
- Bug #690586 - fixed log() function to work with negative data
- Bug #684563 - fixed carbon-cache.py —config parameter
- Bug #660861 - fixed nonNegativeDerivative() math for wrapping counters
- Bug #591948 - fixed X-axis labeling that was inaccurate in some situations
- Bug #595652 - fixed bug preventing clustering from working with default settings.py
- Bug #542090 - fixed Y-axis labeling issue for large values with small variance
- Dozens more…
The best part is, the work in this release has continued to be largely a community effort. Almost all bugs that got fixed were reported from users and the vast majority have been fixed because of contributed patches and highly detailed bug reports. In other words, this ain't a one-man show! Thanks to everyone who has contributed code, bug reports, documentation, questions, and answers.
In the interest of not being incredibly wrong again, I will refrain from putting a date on when the next Graphite release will be out. But it will not be another year, that's for sure… Several projects I have to do for work in the coming months are going to involve major enhancements to Graphite, unlike this past year during which I've really only worked on it in my spare time. Thanks again to everyone and happy new year!
Still kickin' - 12/7/10
So the great documentation release of March never materialized, nor did the monthly release cycle last more than one month. All I can say is that my intentions are good, but my follow-through isn't quite what it used to be (at least when it comes to spare-time projects). Fortunately, Graphite has remained a fairly active project in the absence of me posting any new announcements. Many new features have been developed and bugs have been fixed, but alas no new release since February. Today I've decided to put an end to the hiatus and start working on putting 0.9.7 together. I am committing to getting it out the door before the new year. Nothing major is changing, just the culmination of a year of small-ish new features and bug fixes.
There are some cool things coming down the pipe though. A while back I posted on the graphite-dev mailing list about a new database format that will allow datapoints for a single metric to be sharded across multiple servers (currently all the datapoints for a given metric must be colocated). That project got postponed for a few months but is back on my radar now. It will not be in 0.9.7 though.
Some more cool things are on the way too. Per a project at work I am tasked with implementing an interactive graph UI in the coming months. Lucio Torre has already implemented a branch using flot, https://code.launchpad.net/~lucio.torre/graphite/add-flot/+merge/39768 so that is something to check out.
Last but not least, for the past few months I have been working on contributing a chapter for a book on open source software architecture. The chapter covers the architecture of Graphite and explains some of the design choices and scalability factors. Hopefully it will be out Q1-Q2 2011, I'll be sure to announce when it is available.
February Release - 0.9.6 - 2/26/10
This has probably been the most active month of Graphite development since the project was open sourced. Lots of community members have contributed code and ideas to help move Graphite forward. I'm really excited about this, the project is gaining momentum and I hope we can keep that up by continuing with the new monthly release cycle. To give credit where it is due, here is a list of this month's most active users and what they've been working on (in no particular order):
- Lucio Torre - AMQP support
- jdugan - beautification of the Y-axis labels via the yUnitSystem option
- Nick Leskiw - the YAxis=right rendering option
- Kraig Amador - tons of rendering options/functions such as yLimit, timeShift(), log(), sumSeriesWithWildcard(), new filtering functions, and much more! (Kraig you're the man!)
- Arthur Gautier - debian packaging
- Elliot Murphy - packaging, inclusion in Ubuntu
- fp - RHEL / CentOS RPM packaging
- and many more…
Thanks to everyone who has gotten involved with Graphite, your support helps motivate others (especially me).
Many of these new features are really great but unfortunately undocumented, but the good news is that my focus for March is going to be 100% on *documentation*. There may not be an actual code release in March but I hope to get a substantial amount of documentation written right here on this wiki. Stay tuned.
PS. I'm going to try and update the Roadmap page once a month as well.
New Year, New Release, New Strategy - 1/4/10
It's hard to believe it's been an entire year since the last release of Graphite. This just goes to show how good I am at procrastination. After taking a look at the old 0.9.4 release I can safely say that the new 0.9.5 release is very significant. Here are the biggest changes:
- Graphite now supports federated storage (for better scalability)
- Carbon was completely rewritten using Twisted (much cleaner, more configurable, and more fault tolerant)
- The installation process now uses distutils (finally!)
- Graphite, Carbon, and Whisper are now three separate packages (for more flexible deployment)
- Graphite's browser UI fully migrated to pure ExtJS 3.0 (cleaner code, less bugs)
- Many many bug fixes as always
I'd like to thank everyone in the community who has been using graphite and contributing to the project. I don't usually do new year's resolutions, but I've got a good idea for one this year. I really want to get away from infrequent huge releases like this one and get back to very frequent small releases in the true spirit of open source. So my resolution is to release *something* once a month. It will probably usually just be bug fixes, or perhaps some much needed documentation. So look forward to something new in February!
Also to help keep everyone in the loop on upcoming plans I've started a new Roadmap page. There are a lot of cool things coming up so stay tuned.
Major enhancements in trunk, new 0.9.5 release coming soon - 10/25/09
I've been busy working on Graphite for the past several weeks and now a bunch of code is getting merged back into trunk. More details on the latest changes are available here. The short story is that Graphite is getting more scalable in the backend, more usable in the frontend, and more robust overall. Lots of problems fixed, lots of messes cleaned up, lots of groundwork laid for the future, this should be a good release. We'll be testing for the next couple weeks, after that we'll release 0.9.5, and then some new documentation (with brief video tutorials!) before Thanksgiving.
Major enhancements in the works - 9/4/09
The last release of Graphite is getting a bit stale and a lot of bug fixes have made their way into trunk over the last several months. Thanks to everyone who keeps asking questions, submitting bug reports, and sending in patches. The active community is what has kept this project from stagnating over the past several months as I changed jobs back in March and haven't been able to refocus my efforts on Graphite until recently. There are a lot of exciting things in the works! Currently I am putting the finishing touches on a complete rewrite of carbon that will support federated storage. The new carbon is built on Twisted and many of the administrative annoyances of the previous multi-process carbon have been addressed. On the UI front, I have removed the prototype.js dependency from the Composer UI and upgraded to ExtJS 3.0 (which now comes bundled with Graphite instead of requiring a separate download). The look and feel have not changed significantly but many bugs have been fixed as a result of this. The last major change I have planned for the next release is to migrate the installation process to use distutils. I hope to have the release ready in a month or so, in the mean time keep the bug reports coming! Thanks.
Finally, a new release… 0.9.4 - 1/30/09
It's been a good 6 months since the last release. Not much has changed aside from a few minor enhancements and some good bug fixes, unfortunately I've not had nearly as much time as I'd like to dedicate to working on Graphite. Regardless, it is getting more mature slowly but surely. In the next few months I may be in a better position to get more real work done on it, but we shall see. For now I'd just like to thank everyone who has given me great questions and bug reports, your feedback is what keeps this project moving. Thanks.
New Release 0.9.3 - 7/16/08
This release is an incremental improvement over 0.9.2, including lots of bug fixes, major enhancements to the installer, and several new handy scripts. Thanks to everyone who submitted bug reports and questions. The next few Graphite releases will continue to focus on quality rather than new features. In particular, 0.9.4 will include a re-write of the carbon backend, which will be much simpler and easier to administer and troubleshoot. I am also working on porting lots of internal documentation to this wiki. My goal is to have a 1.0 release by the end of the year, which must be well-documented, easy to deploy, easy to troubleshoot, and of course as bug-free as possible. If there is time a new feature or two might make it in, but this is not the primary focus.